I just had this issue with rclone. Had to generate a new token. First time it has happened.
I got 4TB, some times it fails, and I got to regenerate tokens for all my backups
Last successful backup: 08/10/2022 (took 08:30:26)
Next scheduled run: Wednesday at 2:30 PM
Source: 4.16 TB
Backup: 4.10 TB / 8 Versions
Majority of that time is spent on deleting unwanted files - even when nothing gets deleted (as it shouldn’t)
Next canary when? The last one is in June…
Mars 2027 
Maybe… or hmm¿
How can we trigger canary build / who can trigger it?
There’s a fix in the code for months, but nobody to press the button on GitHub…
I’m only following this topic loosely because I can’t do much, but let’s try to note some key activities.
Canary is pretty much never scheduled. The release manager just finds some time, and one happens. You can look at the releases page you posted to see history, but help here seems to be very low lately because of lack of time from the key people, but not enough new expert volunteers to make up for it…
Even less expert people to help on code, test, docs, etc. are not plentiful. Please all, do what you can.
Presumably (if not, what?) referring to latest post here by @albertony referencing two pull requests.
The June 23 Duplicati one cited there was committed to master on Aug 7 with the following comment:
Based on the forum discussions and tests, this seems to fix the issue.
without mention of oauth-handler pull request. The same has-no-time person does oauth and releases, which historically happen when enough changes accumulate to make it worthwhile. There are four now.
GitHub release announcement has a link that one can click to see what’s been done since that release.
Several people have compiled the change to test and/or to allow others to run. Thanks to all for helping.
Unfortunately things are not sounding that solid even on new code, and even rclone breaks sometimes.
Still, there seems to be a bit of improvement. Does that sound like an accurate summary of posts here?
1 Like
Unfortunately things are not sounding that solid even on new code, and even rclone breaks sometimes.
Still, there seems to be a bit of improvement. Does that sound like an accurate summary of posts here?
Yes, that seems accurate. Backups that have a small amount of data to upload seem to be fine. Backups that have a large amount of data seem to still fail. When they do fail, they seem to fail at whatever the last steps of the backup are (usually around the part where it says: Deleting unwanted files …). Let me know if you guys need more specific logs to help with the cause.
1 Like
I’m not sure if this helps, but most of the time that it fails, I have to repair 1 file. I’ve posted the logs here JIC it’s of any help:
Failed: Found 1 files that are missing from the remote storage, please run repair
Details: Duplicati.Library.Interface.RemoteListVerificationException: Found 1 files that are missing from the remote storage, please run repair
at Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList(BackendManager backend, Options options, LocalDatabase database, IBackendWriter log, IEnumerable1 protectedFiles) at Duplicati.Library.Main.Operation.BackupHandler.PreBackupVerify(BackendManager backend, String protectedfile) at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__20.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at CoCoL.ChannelExtensions.WaitForTaskOrThrow(Task task) at Duplicati.Library.Main.Controller.<>c__DisplayClass14_0.<Backup>b__0(BackupResults result) at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action1 method)
Log data:
2022-09-06 04:15:59 -07 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingFile]: Missing file: duplicati-i7d9fc80e7cd0424e8f89f50847ffb618.dindex.zip.aes
2022-09-06 04:15:59 -07 - [Error-Duplicati.Library.Main.Operation.FilelistProcessor-MissingRemoteFiles]: Found 1 files that are missing from the remote storage, please run repair
2022-09-06 04:15:59 -07 - [Error-Duplicati.Library.Main.Operation.BackupHandler-FatalError]: Fatal error
Duplicati.Library.Interface.RemoteListVerificationException: Found 1 files that are missing from the remote storage, please run repair
at Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList(BackendManager backend, Options options, LocalDatabase database, IBackendWriter log, IEnumerable`1 protectedFiles)
at Duplicati.Library.Main.Operation.BackupHandler.PreBackupVerify(BackendManager backend, String protectedfile)
at Duplicati.Library.Main.Operation.BackupHandler.d__20.MoveNext()
What I’m realizing is that this topic on the 401 error drifted into the 500 error (roughly 50/50 based on searching for mentions in posts), then this latest post doesn’t seem to have any HTTP error mentions.
Disable automatic use of v2 authid for Jottacloud #4779 which got committed Aug 7 awaiting Canary release (with some unofficial builds for testing) is mentioning 500 and 400, and people saying fix didn’t work completely seem to be seeing 500 even after the fix (although what’s run is sometimes unstated).
So at the moment Jottacloud issues seem rather numerous, but I wonder if some are just generic ones such as missing remote dindex files. Known causes for that include failures during a compact which is part of “Deleting unwanted files”. You’d have to either remember error history or set up a log file to log.
Error during compact forgot a dindex file deletion, getting Missing file error next run. #4129
The log-file=<path> log-file-log-level=retry clue for this is if the named missing file had logged its delete.
1 Like
That’s the error I got yesterday.
I use the latest canary with the .dlls which include the fixes from above.
Everything went smooth for several weeks until yesterday.
The backup job was done, but a lot of files had to be deleted. Which means download, repack, upload. After ~12 hours the job failed.
Duplicati.Library.Interface.UserInformationException: Autorisation mit dem OAuth Service ist fehlgeschlagen: Server error. Wenn das Problem weiter besteht, versuche ein neues AuthID Token zu generieren von: Duplicati OAuth Handler —> System.Net.WebException: The remote server returned an error: (500) Internal Server Error.
I created a new token, restarted the job and it failed again after ~4 hours.
On the third try the job finished properly.
Could it be that there are too many requests to Jottacloud which fails the token?
2 Likes
I had the same behavior the last days.
Because sometimes the token expired, I had to change it into a new one in 7 jobs. Because it’s no big fun, I wanted to try just one job for all and deleted some old versions to gather enough free cloud space. Regarding the token lifetime a big mistake!
Uploading large backups, deleting versions (huge amount of data) or repairing databases, all of them failed several times with oauth service errors until they were finally finished.
Now smaller update backups are working fine. At least it seems so.
(with 2.0.6.104_canary_2022-06-15 for Windows)
(Of course I went back to separate backup jobs!)
1 Like
Because that does not have the latest code fix attempt, did you install the unofficial DLLs as well?
Let me attempt a more thorough review of who it seemed to help. Anyone can update their status:
(but I’ll update the below from the posts – seems to be looking better as people pick up the patch)
Y @wmrch
Y @FreaQ
Y @Duplicaterr
Y @dukrat (partial)
Y @waLIEN (partial, --asynchronous-concurrent-upload-limit=1 helped some, big backups worse)
N @SirTerrific (unknown Duplicati)
Y @jankkm
Y @Dominic-Schaefer
Y @SulfurRacoon (but saw at least one different error)
? @shalmirane (unknown Duplicati presumably without latest changes because release is sought)
@waLIEN did an informal summary of the above here, but there’s uncertainty about some versions.
You don’t have the latest fix unless you did install from file (so not Duplicati autoupdate) to 2.0.6.104 followed by replacing the unofficial DLLs. In an autoupdate situation the first install is just a launcher searching for installed updates and running them. Patching that launcher won’t help Jottacloud code. Patching the update will be rejected because files are validated. You must run patched base version.
About → System info can show BaseVersionName and ServerVersionName, if you need that detail.
It would be nice if people who saw no improvement above can report on a 2.0.6.104 + patched base.
The better we can feel about the latest code fix, the better the chances release manager will release.
I’d have to ask because he doesn’t follow the forum much, but I want some good stats to support me.
If anybody wants to go into the (vaguely-defined) release manager role, there’s an opening available.
Now I did.
So I’m using 2.0.6.104_canary_2022-06-15 for Windows (updated via GUI) with the 3 DLLs (originals in “C:\Program Files\Duplicati 2” overwritten) and the option asynchronous-concurrent-upload-limit=1.
I have no issues for the moment. Big thanks!
I’ll post, when/if the next error appears.
If autoupdated means updated to 2.0.6.104 without an install from .msi file, the patch isn’t in use.
You need to get your “C:\Program Files\Duplicati 2” itself on 2.0.6.104 plus the unofficial patches.
In addition to checking About → System info to find your versions, you can read changelog.txt
Please make sure your base install (in Program Files folder) is 2.0.6.4 plus the unofficial patches.
I think an autoupdate 2.0.6.104 (if present) won’t pose a problem, but base needs to be as above.
I’m sorry, my fault, I meant, that I updated it via GUI at “About”.
System Info is saying base version is still 2.0.5.111_canary_2020-09-26.
I just installed duplicati-2.0.6.104_canary_2022-06-15-x64.msi now and in addition overwrote the 3 DLLs again.
Now it’s both the same version number:
- ServerVersionName : - 2.0.6.104_canary_2022-06-15
- BaseVersionName : 2.0.6.104_canary_2022-06-15
1 Like
Good news at least for me:
Today I had 6 backup jobs done in less than 1 h each and a very big one done in 14 h (eactly 13:59:50) without any errors (though there was an automatic router reboot in the night)!
Because the upload speed seems limited by jottacloud I think, I have only 1,2 to 1,5 MB/s. (My connection usually supports 51 Mbit/s = 6,4 MB/s.) That means the big backup was about 60-75 GB.
Since the login changes at jottacloud I was never able to upload such big data without errors.
Thank you all very much!
Correct, I’m running the latest canary available.
I was never able to get that dev virtual environment running in VirtualBox, I guess they were never meant to run on Home versions of windows host; as no matter what I disabled from HyperV, it never finished booting to the login screen.
Just for information:
It is not the same error, but since the error seems to be caused by jottacloud, this log may be useful here:
System.AggregateException: Mindestens ein Fehler ist aufgetreten. ---> System.AggregateException: In die Übertragungsverbindung können keine Daten geschrieben werden: Eine vorhandene Verbindung wurde vom Remotehost geschlossen.(...)
(…) —> System.IO.IOException: In die Übertragungsverbindung können keine Daten geschrieben werden: Eine vorhandene Verbindung wurde vom Remotehost geschlossen. —> System.Net.Sockets.SocketException: Eine vorhandene Verbindung wurde vom Remotehost geschlossen
bei System.Net.Sockets.Socket.EndSend(IAsyncResult asyncResult)
bei System.Net.Sockets.NetworkStream.EndWrite(IAsyncResult asyncResult)
— Ende der internen Ausnahmestapelüberwachung —
bei CoCoL.AutomationExtensions.d__101.MoveNext() --- Ende der Stapelüberwachung vom vorhergehenden Ort, an dem die Ausnahme ausgelöst wurde --- bei System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() bei System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) bei Duplicati.Library.Main.Operation.BackupHandler.<FlushBackend>d__19.MoveNext() --- Ende der Stapelüberwachung vom vorhergehenden Ort, an dem die Ausnahme ausgelöst wurde --- bei System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() bei System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) bei Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__20.MoveNext() --- Ende der internen Ausnahmestapelüberwachung --- bei Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__20.MoveNext() --- Ende der internen Ausnahmestapelüberwachung --- bei CoCoL.ChannelExtensions.WaitForTaskOrThrow(Task task) bei Duplicati.Library.Main.Controller.<>c__DisplayClass14_0.<Backup>b__0(BackupResults result) bei Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action1 method)
bei Duplicati.Library.Main.Controller.Backup(String[] inputsources, IFilter filter)
bei Duplicati.Server.Runner.Run(IRunnerData data, Boolean fromQueue)
—> (Interne Ausnahme #0) System.AggregateException: In die Übertragungsverbindung können keine Daten geschrieben werden: Eine vorhandene Verbindung wurde vom Remotehost geschlossen. —> System.IO.IOException: In die Übertragungsverbindung können keine Daten geschrieben werden: Eine vorhandene Verbindung wurde vom Remotehost geschlossen. —> System.Net.Sockets.SocketException: Eine vorhandene Verbindung wurde vom Remotehost geschlossen
bei System.Net.Sockets.Socket.EndSend(IAsyncResult asyncResult)
bei System.Net.Sockets.NetworkStream.EndWrite(IAsyncResult asyncResult)
— Ende der internen Ausnahmestapelüberwachung —
bei CoCoL.AutomationExtensions.d__101.MoveNext() --- Ende der Stapelüberwachung vom vorhergehenden Ort, an dem die Ausnahme ausgelöst wurde --- bei System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() bei System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) bei Duplicati.Library.Main.Operation.BackupHandler.<FlushBackend>d__19.MoveNext() --- Ende der Stapelüberwachung vom vorhergehenden Ort, an dem die Ausnahme ausgelöst wurde --- bei System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() bei System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) bei Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__20.MoveNext() --- Ende der internen Ausnahmestapelüberwachung --- bei Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__20.MoveNext() ---> (Interne Ausnahme #0) System.IO.IOException: In die Übertragungsverbindung können keine Daten geschrieben werden: Eine vorhandene Verbindung wurde vom Remotehost geschlossen. ---> System.Net.Sockets.SocketException: Eine vorhandene Verbindung wurde vom Remotehost geschlossen bei System.Net.Sockets.Socket.EndSend(IAsyncResult asyncResult) bei System.Net.Sockets.NetworkStream.EndWrite(IAsyncResult asyncResult) --- Ende der internen Ausnahmestapelüberwachung --- bei CoCoL.AutomationExtensions.<RunTask>d__101.MoveNext()
— Ende der Stapelüberwachung vom vorhergehenden Ort, an dem die Ausnahme ausgelöst wurde —
bei System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
bei System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
bei Duplicati.Library.Main.Operation.BackupHandler.d__19.MoveNext()
— Ende der Stapelüberwachung vom vorhergehenden Ort, an dem die Ausnahme ausgelöst wurde —
bei System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
bei System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
bei Duplicati.Library.Main.Operation.BackupHandler.d__20.MoveNext()<—
—> (Interne Ausnahme #1) System.AggregateException: Mindestens ein Fehler ist aufgetreten. —> System.IO.IOException: In die Übertragungsverbindung können keine Daten geschrieben werden: Eine vorhandene Verbindung wurde vom Remotehost geschlossen. —> System.Net.Sockets.SocketException: Eine vorhandene Verbindung wurde vom Remotehost geschlossen
bei System.Net.Sockets.Socket.EndSend(IAsyncResult asyncResult)
bei System.Net.Sockets.NetworkStream.EndWrite(IAsyncResult asyncResult)
— Ende der internen Ausnahmestapelüberwachung —
bei CoCoL.AutomationExtensions.d__101.MoveNext() --- Ende der Stapelüberwachung vom vorhergehenden Ort, an dem die Ausnahme ausgelöst wurde --- bei System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() bei System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) bei Duplicati.Library.Main.Operation.BackupHandler.<FlushBackend>d__19.MoveNext() --- Ende der Stapelüberwachung vom vorhergehenden Ort, an dem die Ausnahme ausgelöst wurde --- bei System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() bei System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) bei Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__20.MoveNext() --- Ende der internen Ausnahmestapelüberwachung --- ---> (Interne Ausnahme #0) System.IO.IOException: In die Übertragungsverbindung können keine Daten geschrieben werden: Eine vorhandene Verbindung wurde vom Remotehost geschlossen. ---> System.Net.Sockets.SocketException: Eine vorhandene Verbindung wurde vom Remotehost geschlossen bei System.Net.Sockets.Socket.EndSend(IAsyncResult asyncResult) bei System.Net.Sockets.NetworkStream.EndWrite(IAsyncResult asyncResult) --- Ende der internen Ausnahmestapelüberwachung --- bei CoCoL.AutomationExtensions.<RunTask>d__101.MoveNext()
— Ende der Stapelüberwachung vom vorhergehenden Ort, an dem die Ausnahme ausgelöst wurde —
bei System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
bei System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
bei Duplicati.Library.Main.Operation.BackupHandler.d__19.MoveNext()
— Ende der Stapelüberwachung vom vorhergehenden Ort, an dem die Ausnahme ausgelöst wurde —
bei System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
bei System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
bei Duplicati.Library.Main.Operation.BackupHandler.d__20.MoveNext()<—
<—
<—
The error appeared with Duplicati - 2.0.6.104_canary_2022-06-15 (installte MSI + 3 DLLs) while doing a scheduled backup.
The Oauth token is still valid.
Other later scheduled backups had no errors.
A manual retry did fine.
To add to the fun, I just deployed the code in PR/10 from duplicati-oauth-handler to https://duplicati-oauth-handler-beta.appspot.com/ again. I have not yet tried the latest Duplicati code because I did not know of this problem when I started my failed backup again.
One puzzling question I have, and maybe @Albertony can chime in here, but why are we disabling oauth v2? Without reading through the code, but instead reading through the comments, it seems a quick short-term fix was to ignore the V2 tokens that come back from Jotta and instead continue to use the V1 tokens. This seems like a waiting timebomb when Jotta flat-out refuses V1 tokens and/or refuses to issue new Cli tokens that support V1. What am I missing here?
Now I am running the unlikely combination of an old Duplicati that does not have the code fix (presumably in the 3 DLLs mentioned) and the oauthh handler fixes. This combination would presumably force my Jotta to use V1 tokens because the oauth handler is refusing to hand out v2 tokens. Is that kindda right?
Unless there is some horrible reason why we cannot use the v2 tokens, I’d say we should try to implement v2 before Jotta removes support for v1.
Quick answer, of the top of my head (may add/correct more later when I have time):
Correct. With the oauth fix the Duplicati fix is not necessary - i.e. the fix which is in master, but no releases yet, but is also in the 3 dlls for simple patching. They both basically do the same thing, just from separate sides: Oauth service not sending v2 tokens vs Duplicati not accepting v2 tokens.
V2 is simply a concept in the Duplicati OAuth handler where it returns the actual refresh token back to Duplicati instead of storing the refresh token internally and return a lookup key to it (the authid). So it basically has nothing to do with Jottacloud or OAuth at all.