Backup test errors - The file duplicati-(...).dblock.zip.aes was downloaded and had size X but the size was expected to be Y

Hi,

During the test phase of my backups I regularly get errors like this:
2021-08-08 01:10:16 +00 - [Error-Duplicati.Library.Main.Operation.TestHandler-FailedToProcessFile]: Failed to process file duplicati-b3bbe034108a54519896c09910890585f.dblock.zip.aes
System.Exception: The file duplicati-b3bbe034108a54519896c09910890585f.dblock.zip.aes was downloaded and had size 387973120 but the size was expected to be 433959789

2021-08-08 08:14:12 +00 - [Error-Duplicati.Library.Main.Operation.TestHandler-FailedToProcessFile]: Failed to process file duplicati-bf6150169135a48619bf90fe86a6881af.dblock.zip.aes
System.Exception: The file duplicati-bf6150169135a48619bf90fe86a6881af.dblock.zip.aes was downloaded and had size 31390765 but the size was expected to be 52362285

I usually solve these by running a purge-broken-files + restarting the backup, but I would like to know the cause of this.
Blocksize is 100 MB and DBLOCK size is 500 MB. I run the backup weekly with a test percentage of 5% (which boils down to the double but I already started a topic on that one :wink: ). It concerns a backup operation to Jottacloud. The hardware is a Synology 216±II with Duplicati running in a Docker-container (version is Duplicati - 2.0.6.3_beta_2021-06-17).

If a dblock filesize gets less than expected I would guess something is wrong with Jottacloud, but a quick search here learned me that there are no errors reported on that one yet… so I would like to check if there are any other issues?

Thanks in advance :wink:

Do you have any ways (maybe through Jottacloud) to see that file’s date and size and then download it?
It would be helpful to see if it’s new for that backup, and what size some other downloader is able to get.

By the way, 387973120 is 17200000 in hexadecimal (by Windows Calculator). It might mean something.
You can compare similar messages to see if possible truncation to a binary-even size value is common.

If you want to try a download with a Duplicati downloader, Duplicati.CommandLine.BackendTool.exe get
can do it. Use a URL similar to what Export As Command-line has, except add the desired filename to it.

Thanks for your answer. Before purging and reuploading the files I made a copy of them in another folder on Jottacloud. So I was able to find back the original files. I did what you asked (checked the size and downloaded them). The result is very surprising!
After downloading the original files, the local filesize was exactly the expected size. So, for the first file, it was 433959789 instead of 387973120, likewise for the second one.
In the Jottacloud web interface (from which I also downloaded the files) I can see that the displayed size (which only shows in MB) is in line with the expected sizes as well.
So it seems like Duplicati somehow misreads the filesize and hence considers the test as failed - most probably wrongly.

What could be the cause of this?

Or it correctly read the filesize, but that file didn’t completely download due to lack of /tmp space (did you check?) although that seems it would cause more issues on uploads, or because the download failed, or because it worked to the end of file but was cut short because Jottacloud hadn’t fully stored that file yet…
That’s why I asked for that file’s date, so we could know whether it was uploaded shortly before download.

Unfortunately, it sounds like this is a regular problem. Bad for backup, but good to try to look for the trends. Keeping an eye on the incorrect file size would be useful to see if it’s always a suspiciously even hex size. Although the first file size was extremely even viewed in hex, second was not, but this is just two samples.

Assuming you downloaded with Duplicati’s BackendTool, that means that that tool got good file at that time, however it’s hard to tell what the actual backup got, short of trying to copy files out of /tmp during download.

This message is related to downloading, which happens at several places (more later). Size error is below:

It might take some searching through the job logs to find them, but Compacting files at the backend also downloads files (during the Deleting unwanted files status bar step). It puts results of that in the log, although some forms of compact don’t do downloading (they delete). Still, I wonder how download goes? Typically, these downloads are old files which have been around long enough to have now-wasted space when backup versions get deleted, thereby potentially removing need to keep some changes they record.

You can get a log of the exact timestamped history of files to look at later when files give error messages.
log-file=<path> at log-file-log-level=retry will do it, and also show retries. Also look in regular logs for value "RetryAttempts" in Complete log. Unfortunately, a regular log won’t be made when backup errors out.

You can actually see the backend file history in Job → Show log → Remote, but paging through it is hard, but there’s a chance that when a backup errors, the files with downloaded errors will be among the latest.

The same log will have a list before the test download, so you can see what size Jottacloud “claimed”.
Example end of a backup, including the default sample size of 1 set of files – a dlist, dindex, and dblock:

image

Click on a line to get more information. In the case of a list you can see things like this. Times are UTC:

{“Name”:“duplicati-b5704070b9e0b4116b057f3d3f6043615.dblock.zip.aes”,“LastAccess”:“2021-08-18T19:42:20.733Z”,“LastModification”:“2021-08-18T19:42:20.733Z”,“Size”:1514045,“IsFolder”:false},

You can also test all of your files if you like, to see if they download currently and have expected size and data. This can of course take awhile if you have a large backup or slow download speed. To run this, see
The TEST command which you can run from GUI Commandline or from shell Export As Command-line. Either way, you start with a backup command and edit the format into what you want (in this case, test).

Hi,

  • About the tmp/ location: is there a way to check a lack of disk space? Duplicati is running in a Docker container, AFAIK there are no limitations in the disk space it can use (I might be wrong on this one?). The NAS has ± 2,2 TB of free disk space which is way more than the total backup set.
  • File modification dates: For duplicati-b3bbe034108a54519896c09910890585f.dblock.zip.aes last modified date is 11 May 2021, for duplicati-bf6150169135a48619bf90fe86a6881af.dblock.zip.aes it is 13 Nov 2020. For both the error message was on 8 Aug 2021 so no direct relationship there.
  • I did the download via a webbrowser from the Jottacloud web interface. If it makes a difference in troubleshooting I can use the Backendtool instead, just tell me.
  • About the compacting: it doesn’t seem to me that it concerns a compacting issue. The data that is impacted is data that is still present on the NAS (so not deleted) and I want to keep a backup of it on Jottacloud. Also the data is highly immutable. None of the impacted data has been changed shortly before taking the backup (actually, nothing is changed since 2019. And backup operations happens weekly…)
  • For one of the failing backups I have “RetryAttempts”: 43 ; for the other one it is “RetryAttempts”: 58.
  • For the exact timestamped log: I set up the logging as you requested. Let’s check this for the next times (backups happen on weekly basis)…

Probably need a Docker user to say whether /tmp is in the Docker (in which case it’s limited) or mapped through to host (in which case it’s likely less limited). If Docker has a df, then it can probably show space.

You would have to test if it makes a difference. I’m sure the code is different from the Jottacloud web UI’s.
It should be similar to the regular Duplicati code, which (if I followed right) got short download on files that had been there awhile, and which other tools download in total. I don’t know why that would be happening.

A wrong size download will do a retry, and it’s limited by number-of-retries per download attempt. Question is do you see retries when everything is working? Can’t tell at the moment, just know a total. Logs will help.

Hi again,

Thanks for your patient answers…

All backups have run this weekend without a single error. I am keeping the logs for future reference. For the next backups logs will also be taken. If an error occurs again they will come in handy (and I’ll be back…)

For the check on /tmp space inside the Docker, I did a df in the Docker container terminal with these results:

	root@duplicati-duplicati12:/# df -h /tmp
	Filesystem      Size  Used Avail Use% Mounted on
	/dev/vg1000/lv  3.5T  1.4T  2.2T  39% /

I am by no means a Docker expert, but my guess would be that this means that the /tmp inside Docker has more than 2 TB space left.

For the use of the backend tool I am really puzzled. I run the backend tool for downloading (in this case I am doing that with one of the “parked” files that failed before), but even when giving correct path and files, I get the error “Command failed: The requested file does not exist”. Even for filepaths that are clearly listed when I am doing a list… See hereunder:

	root@duplicati-duplicati12:/opt/duplicati# mono Duplicati.CommandLine.BackendTool.exe get "jottacloud://backup/nas_docker/v/f2/restore_tmp/duplicati-b3bbe034108a54519896c09910890585f.dblock.zip.aes?auth-username=(USERNAME)&auth-password=(PASSWORD)" file.dblock.zip.aes
	Command failed: The requested file does not exist

	root@duplicati-duplicati12:/opt/duplicati# mono Duplicati.CommandLine.BackendTool.exe list "jottacloud://backup/nas_docker/v/f2/restore_tmp/?auth-username=(USERNAME)&auth-password=(PASSWORD)"
	Name	Dir/File	LastChange	Size
	(Several files - Scrubbed)
	duplicati-b3bbe034108a54519896c09910890585f.dblock.zip.aes	File	05/11/2021 17:34:33	413.86 MB
	(Several files - Scrubbed)

Either I am doing something really wrong here, or something else is not going allright.
For safety sake I am assuming the first option - Can you tell me what I did wrong here?

When it says:

Usage: <command> <protocol>://<username>:<password>@<path> [filename]
Example: LIST ftp://user:pass@server/folder

I think it means give the destination folder URL (as you did for list command), and filename (if used).
Perhaps you’re trying to download into file.dblock.zip.aes. I think you just get the same name as remote.
Be careful where you download to. Safest would be to cd somewhere safe, and use full command path.

That’s the solution indeed. Folder- and filenames are in 2 separate parameters and the local file has the same name as the remote one.

So I was able to download the so called “wrong-sized” file (the one I parked before purging and redoing the backup) with the command line tool and compared it to the one I downloaded from the Jottacloud web interface. The files are identical.

Mystery remains … Logs are still on for next time the error happens. Have you got other ideas meanwhile?

Jottacloud temporary problem? The even-binary-size one seems especially likely to be storage-related.

Second size didn’t show that though, so possibly an undetected network error could cut the file short…
This would be difficult to catch (and I have no evidence it’s the issue) without some network monitoring.

https://usage-reporter.duplicati.com/ shows Jottacloud is little used, so it’s possible code is not perfect.
Code was also reverse-engineered. Although Duplicati doesn’t use the library, see some history at jafs.

There was a 2019 change adding the below. If problems persist, you could see if option set to 1 avoids.

  --jottacloud-threads (Integer): Number of threads for restore operations.
    Number of threads for restore operations. In some cases the download rate
    is limited to 18.5 Mbps per stream. Use multiple threads to increase
    throughput.
    * default value: 4

EDIT:

Above mentions restore, but it affects all downloads I believe, including those done in test and compact.
Possibly the below option is relevant by default, but if you need to tune, threads might be your first target:

  --jottacloud-chunksize (Size): The chunk size for simultaneous downloading.
    The chunk size for simultaneous downloading. These chunks will be held in
    memory, so keep it as low as possible.
    * default value: 5mb

After a while, got errors again!
Error seems to be
The remote server returned an error: (404) Not Found. (full log underneath for this file).
After 5 retries the file is abandoned. I got this for 4 DBLOCK files of the backup.
The files are present and a test restore worked without problems.

What could it be? Just Jottacloud-related? Any ideas?

Full log for this file occurance:

2022-01-09 03:21:53 +00 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-b0ecf865843e2496780512880cabed88d.dblock.zip.aes (481.67 MB)
2022-01-09 03:21:54 +00 - [Retry-Duplicati.Library.Main.BackendManager-RetryGet]: Operation Get with file duplicati-b0ecf865843e2496780512880cabed88d.dblock.zip.aes attempt 1 of 5 failed with message: One or more errors occurred. (The remote server returned an error: (404) Not Found.)
System.AggregateException: One or more errors occurred. (The remote server returned an error: (404) Not Found.) ---> System.Net.WebException: The remote server returned an error: (404) Not Found.
  at System.Net.HttpWebRequest.GetResponseFromData (System.Net.WebResponseStream stream, System.Threading.CancellationToken cancellationToken) [0x00146] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest+AsyncWrapper.GetResponseOrStream () [0x0004d] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest.GetResponse () [0x00044] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest.GetResponseStream () [0x00000] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Backend.Jottacloud+<>c__DisplayClass51_1.<ParallelGet>b__0 () [0x0004a] in <9506f4c8d1134272aa6ea119baff3a59>:0 
  at System.Threading.Tasks.Task`1[TResult].InnerInvoke () [0x0000f] in <d13c8b563008422a8c5aaec0a74089cc>:0 
  at System.Threading.Tasks.Task.Execute () [0x00000] in <d13c8b563008422a8c5aaec0a74089cc>:0 
   --- End of inner exception stack trace ---
  at System.Threading.Tasks.Task.ThrowIfExceptional (System.Boolean includeTaskCanceledExceptions) [0x00011] in <d13c8b563008422a8c5aaec0a74089cc>:0 
  at System.Threading.Tasks.Task`1[TResult].GetResultCore (System.Boolean waitCompletionNotification) [0x0002b] in <d13c8b563008422a8c5aaec0a74089cc>:0 
  at System.Threading.Tasks.Task`1[TResult].get_Result () [0x0000f] in <d13c8b563008422a8c5aaec0a74089cc>:0 
  at Duplicati.Library.Backend.Jottacloud.ParallelGet (System.String remotename, System.IO.Stream stream) [0x000b7] in <9506f4c8d1134272aa6ea119baff3a59>:0 
  at Duplicati.Library.Backend.Jottacloud.Get (System.String remotename, System.IO.Stream stream) [0x00009] in <9506f4c8d1134272aa6ea119baff3a59>:0 
  at Duplicati.Library.Main.BackendManager.coreDoGetPiping (Duplicati.Library.Main.BackendManager+FileEntryItem item, Duplicati.Library.Interface.IEncryption useDecrypter, System.Int64& retDownloadSize, System.String& retHashcode) [0x00259] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.BackendManager.DoGet (Duplicati.Library.Main.BackendManager+FileEntryItem item) [0x002ff] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.BackendManager.ThreadRun () [0x000ff] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
---> (Inner Exception #0) System.Net.WebException: The remote server returned an error: (404) Not Found.
  at System.Net.HttpWebRequest.GetResponseFromData (System.Net.WebResponseStream stream, System.Threading.CancellationToken cancellationToken) [0x00146] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest+AsyncWrapper.GetResponseOrStream () [0x0004d] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest.GetResponse () [0x00044] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest.GetResponseStream () [0x00000] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Backend.Jottacloud+<>c__DisplayClass51_1.<ParallelGet>b__0 () [0x0004a] in <9506f4c8d1134272aa6ea119baff3a59>:0 
  at System.Threading.Tasks.Task`1[TResult].InnerInvoke () [0x0000f] in <d13c8b563008422a8c5aaec0a74089cc>:0 
  at System.Threading.Tasks.Task.Execute () [0x00000] in <d13c8b563008422a8c5aaec0a74089cc>:0 <---

2022-01-09 03:21:54 +00 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Retrying: duplicati-b0ecf865843e2496780512880cabed88d.dblock.zip.aes (481.67 MB)
2022-01-09 03:22:04 +00 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-b0ecf865843e2496780512880cabed88d.dblock.zip.aes (481.67 MB)
2022-01-09 03:22:05 +00 - [Retry-Duplicati.Library.Main.BackendManager-RetryGet]: Operation Get with file duplicati-b0ecf865843e2496780512880cabed88d.dblock.zip.aes attempt 2 of 5 failed with message: One or more errors occurred. (The remote server returned an error: (404) Not Found.)
System.AggregateException: One or more errors occurred. (The remote server returned an error: (404) Not Found.) ---> System.Net.WebException: The remote server returned an error: (404) Not Found.
  at System.Net.HttpWebRequest.GetResponseFromData (System.Net.WebResponseStream stream, System.Threading.CancellationToken cancellationToken) [0x00146] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest+AsyncWrapper.GetResponseOrStream () [0x0004d] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest.GetResponse () [0x00044] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest.GetResponseStream () [0x00000] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Backend.Jottacloud+<>c__DisplayClass51_1.<ParallelGet>b__0 () [0x0004a] in <9506f4c8d1134272aa6ea119baff3a59>:0 
  at System.Threading.Tasks.Task`1[TResult].InnerInvoke () [0x0000f] in <d13c8b563008422a8c5aaec0a74089cc>:0 
  at System.Threading.Tasks.Task.Execute () [0x00000] in <d13c8b563008422a8c5aaec0a74089cc>:0 
   --- End of inner exception stack trace ---
  at System.Threading.Tasks.Task.ThrowIfExceptional (System.Boolean includeTaskCanceledExceptions) [0x00011] in <d13c8b563008422a8c5aaec0a74089cc>:0 
  at System.Threading.Tasks.Task`1[TResult].GetResultCore (System.Boolean waitCompletionNotification) [0x0002b] in <d13c8b563008422a8c5aaec0a74089cc>:0 
  at System.Threading.Tasks.Task`1[TResult].get_Result () [0x0000f] in <d13c8b563008422a8c5aaec0a74089cc>:0 
  at Duplicati.Library.Backend.Jottacloud.ParallelGet (System.String remotename, System.IO.Stream stream) [0x000b7] in <9506f4c8d1134272aa6ea119baff3a59>:0 
  at Duplicati.Library.Backend.Jottacloud.Get (System.String remotename, System.IO.Stream stream) [0x00009] in <9506f4c8d1134272aa6ea119baff3a59>:0 
  at Duplicati.Library.Main.BackendManager.coreDoGetPiping (Duplicati.Library.Main.BackendManager+FileEntryItem item, Duplicati.Library.Interface.IEncryption useDecrypter, System.Int64& retDownloadSize, System.String& retHashcode) [0x00259] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.BackendManager.DoGet (Duplicati.Library.Main.BackendManager+FileEntryItem item) [0x002ff] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.BackendManager.ThreadRun () [0x000ff] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
---> (Inner Exception #0) System.Net.WebException: The remote server returned an error: (404) Not Found.
  at System.Net.HttpWebRequest.GetResponseFromData (System.Net.WebResponseStream stream, System.Threading.CancellationToken cancellationToken) [0x00146] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest+AsyncWrapper.GetResponseOrStream () [0x0004d] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest.GetResponse () [0x00044] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest.GetResponseStream () [0x00000] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Backend.Jottacloud+<>c__DisplayClass51_1.<ParallelGet>b__0 () [0x0004a] in <9506f4c8d1134272aa6ea119baff3a59>:0 
  at System.Threading.Tasks.Task`1[TResult].InnerInvoke () [0x0000f] in <d13c8b563008422a8c5aaec0a74089cc>:0 
  at System.Threading.Tasks.Task.Execute () [0x00000] in <d13c8b563008422a8c5aaec0a74089cc>:0 <---

2022-01-09 03:22:05 +00 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Retrying: duplicati-b0ecf865843e2496780512880cabed88d.dblock.zip.aes (481.67 MB)
2022-01-09 03:22:15 +00 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-b0ecf865843e2496780512880cabed88d.dblock.zip.aes (481.67 MB)
2022-01-09 03:22:16 +00 - [Retry-Duplicati.Library.Main.BackendManager-RetryGet]: Operation Get with file duplicati-b0ecf865843e2496780512880cabed88d.dblock.zip.aes attempt 3 of 5 failed with message: One or more errors occurred. (The remote server returned an error: (404) Not Found.)
System.AggregateException: One or more errors occurred. (The remote server returned an error: (404) Not Found.) ---> System.Net.WebException: The remote server returned an error: (404) Not Found.
  at System.Net.HttpWebRequest.GetResponseFromData (System.Net.WebResponseStream stream, System.Threading.CancellationToken cancellationToken) [0x00146] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest+AsyncWrapper.GetResponseOrStream () [0x0004d] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest.GetResponse () [0x00044] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest.GetResponseStream () [0x00000] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Backend.Jottacloud+<>c__DisplayClass51_1.<ParallelGet>b__0 () [0x0004a] in <9506f4c8d1134272aa6ea119baff3a59>:0 
  at System.Threading.Tasks.Task`1[TResult].InnerInvoke () [0x0000f] in <d13c8b563008422a8c5aaec0a74089cc>:0 
  at System.Threading.Tasks.Task.Execute () [0x00000] in <d13c8b563008422a8c5aaec0a74089cc>:0 
   --- End of inner exception stack trace ---
  at System.Threading.Tasks.Task.ThrowIfExceptional (System.Boolean includeTaskCanceledExceptions) [0x00011] in <d13c8b563008422a8c5aaec0a74089cc>:0 
  at System.Threading.Tasks.Task`1[TResult].GetResultCore (System.Boolean waitCompletionNotification) [0x0002b] in <d13c8b563008422a8c5aaec0a74089cc>:0 
  at System.Threading.Tasks.Task`1[TResult].get_Result () [0x0000f] in <d13c8b563008422a8c5aaec0a74089cc>:0 
  at Duplicati.Library.Backend.Jottacloud.ParallelGet (System.String remotename, System.IO.Stream stream) [0x000b7] in <9506f4c8d1134272aa6ea119baff3a59>:0 
  at Duplicati.Library.Backend.Jottacloud.Get (System.String remotename, System.IO.Stream stream) [0x00009] in <9506f4c8d1134272aa6ea119baff3a59>:0 
  at Duplicati.Library.Main.BackendManager.coreDoGetPiping (Duplicati.Library.Main.BackendManager+FileEntryItem item, Duplicati.Library.Interface.IEncryption useDecrypter, System.Int64& retDownloadSize, System.String& retHashcode) [0x00259] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.BackendManager.DoGet (Duplicati.Library.Main.BackendManager+FileEntryItem item) [0x002ff] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.BackendManager.ThreadRun () [0x000ff] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
---> (Inner Exception #0) System.Net.WebException: The remote server returned an error: (404) Not Found.
  at System.Net.HttpWebRequest.GetResponseFromData (System.Net.WebResponseStream stream, System.Threading.CancellationToken cancellationToken) [0x00146] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest+AsyncWrapper.GetResponseOrStream () [0x0004d] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest.GetResponse () [0x00044] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest.GetResponseStream () [0x00000] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Backend.Jottacloud+<>c__DisplayClass51_1.<ParallelGet>b__0 () [0x0004a] in <9506f4c8d1134272aa6ea119baff3a59>:0 
  at System.Threading.Tasks.Task`1[TResult].InnerInvoke () [0x0000f] in <d13c8b563008422a8c5aaec0a74089cc>:0 
  at System.Threading.Tasks.Task.Execute () [0x00000] in <d13c8b563008422a8c5aaec0a74089cc>:0 <---

2022-01-09 03:22:16 +00 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Retrying: duplicati-b0ecf865843e2496780512880cabed88d.dblock.zip.aes (481.67 MB)
2022-01-09 03:22:26 +00 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-b0ecf865843e2496780512880cabed88d.dblock.zip.aes (481.67 MB)
2022-01-09 03:22:26 +00 - [Retry-Duplicati.Library.Main.BackendManager-RetryGet]: Operation Get with file duplicati-b0ecf865843e2496780512880cabed88d.dblock.zip.aes attempt 4 of 5 failed with message: One or more errors occurred. (The remote server returned an error: (404) Not Found.)
System.AggregateException: One or more errors occurred. (The remote server returned an error: (404) Not Found.) ---> System.Net.WebException: The remote server returned an error: (404) Not Found.
  at System.Net.HttpWebRequest.GetResponseFromData (System.Net.WebResponseStream stream, System.Threading.CancellationToken cancellationToken) [0x00146] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest+AsyncWrapper.GetResponseOrStream () [0x0004d] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest.GetResponse () [0x00044] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest.GetResponseStream () [0x00000] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Backend.Jottacloud+<>c__DisplayClass51_1.<ParallelGet>b__0 () [0x0004a] in <9506f4c8d1134272aa6ea119baff3a59>:0 
  at System.Threading.Tasks.Task`1[TResult].InnerInvoke () [0x0000f] in <d13c8b563008422a8c5aaec0a74089cc>:0 
  at System.Threading.Tasks.Task.Execute () [0x00000] in <d13c8b563008422a8c5aaec0a74089cc>:0 
   --- End of inner exception stack trace ---
  at System.Threading.Tasks.Task.ThrowIfExceptional (System.Boolean includeTaskCanceledExceptions) [0x00011] in <d13c8b563008422a8c5aaec0a74089cc>:0 
  at System.Threading.Tasks.Task`1[TResult].GetResultCore (System.Boolean waitCompletionNotification) [0x0002b] in <d13c8b563008422a8c5aaec0a74089cc>:0 
  at System.Threading.Tasks.Task`1[TResult].get_Result () [0x0000f] in <d13c8b563008422a8c5aaec0a74089cc>:0 
  at Duplicati.Library.Backend.Jottacloud.ParallelGet (System.String remotename, System.IO.Stream stream) [0x000b7] in <9506f4c8d1134272aa6ea119baff3a59>:0 
  at Duplicati.Library.Backend.Jottacloud.Get (System.String remotename, System.IO.Stream stream) [0x00009] in <9506f4c8d1134272aa6ea119baff3a59>:0 
  at Duplicati.Library.Main.BackendManager.coreDoGetPiping (Duplicati.Library.Main.BackendManager+FileEntryItem item, Duplicati.Library.Interface.IEncryption useDecrypter, System.Int64& retDownloadSize, System.String& retHashcode) [0x00259] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.BackendManager.DoGet (Duplicati.Library.Main.BackendManager+FileEntryItem item) [0x002ff] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.BackendManager.ThreadRun () [0x000ff] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
---> (Inner Exception #0) System.Net.WebException: The remote server returned an error: (404) Not Found.
  at System.Net.HttpWebRequest.GetResponseFromData (System.Net.WebResponseStream stream, System.Threading.CancellationToken cancellationToken) [0x00146] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest+AsyncWrapper.GetResponseOrStream () [0x0004d] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest.GetResponse () [0x00044] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest.GetResponseStream () [0x00000] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Backend.Jottacloud+<>c__DisplayClass51_1.<ParallelGet>b__0 () [0x0004a] in <9506f4c8d1134272aa6ea119baff3a59>:0 
  at System.Threading.Tasks.Task`1[TResult].InnerInvoke () [0x0000f] in <d13c8b563008422a8c5aaec0a74089cc>:0 
  at System.Threading.Tasks.Task.Execute () [0x00000] in <d13c8b563008422a8c5aaec0a74089cc>:0 <---

2022-01-09 03:22:26 +00 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Retrying: duplicati-b0ecf865843e2496780512880cabed88d.dblock.zip.aes (481.67 MB)
2022-01-09 03:22:36 +00 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-b0ecf865843e2496780512880cabed88d.dblock.zip.aes (481.67 MB)
2022-01-09 03:22:37 +00 - [Retry-Duplicati.Library.Main.BackendManager-RetryGet]: Operation Get with file duplicati-b0ecf865843e2496780512880cabed88d.dblock.zip.aes attempt 5 of 5 failed with message: One or more errors occurred. (The remote server returned an error: (404) Not Found.)
System.AggregateException: One or more errors occurred. (The remote server returned an error: (404) Not Found.) ---> System.Net.WebException: The remote server returned an error: (404) Not Found.
  at System.Net.HttpWebRequest.GetResponseFromData (System.Net.WebResponseStream stream, System.Threading.CancellationToken cancellationToken) [0x00146] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest+AsyncWrapper.GetResponseOrStream () [0x0004d] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest.GetResponse () [0x00044] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest.GetResponseStream () [0x00000] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Backend.Jottacloud+<>c__DisplayClass51_1.<ParallelGet>b__0 () [0x0004a] in <9506f4c8d1134272aa6ea119baff3a59>:0 
  at System.Threading.Tasks.Task`1[TResult].InnerInvoke () [0x0000f] in <d13c8b563008422a8c5aaec0a74089cc>:0 
  at System.Threading.Tasks.Task.Execute () [0x00000] in <d13c8b563008422a8c5aaec0a74089cc>:0 
   --- End of inner exception stack trace ---
  at System.Threading.Tasks.Task.ThrowIfExceptional (System.Boolean includeTaskCanceledExceptions) [0x00011] in <d13c8b563008422a8c5aaec0a74089cc>:0 
  at System.Threading.Tasks.Task`1[TResult].GetResultCore (System.Boolean waitCompletionNotification) [0x0002b] in <d13c8b563008422a8c5aaec0a74089cc>:0 
  at System.Threading.Tasks.Task`1[TResult].get_Result () [0x0000f] in <d13c8b563008422a8c5aaec0a74089cc>:0 
  at Duplicati.Library.Backend.Jottacloud.ParallelGet (System.String remotename, System.IO.Stream stream) [0x000b7] in <9506f4c8d1134272aa6ea119baff3a59>:0 
  at Duplicati.Library.Backend.Jottacloud.Get (System.String remotename, System.IO.Stream stream) [0x00009] in <9506f4c8d1134272aa6ea119baff3a59>:0 
  at Duplicati.Library.Main.BackendManager.coreDoGetPiping (Duplicati.Library.Main.BackendManager+FileEntryItem item, Duplicati.Library.Interface.IEncryption useDecrypter, System.Int64& retDownloadSize, System.String& retHashcode) [0x00259] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.BackendManager.DoGet (Duplicati.Library.Main.BackendManager+FileEntryItem item) [0x002ff] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.BackendManager.ThreadRun () [0x000ff] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
---> (Inner Exception #0) System.Net.WebException: The remote server returned an error: (404) Not Found.
  at System.Net.HttpWebRequest.GetResponseFromData (System.Net.WebResponseStream stream, System.Threading.CancellationToken cancellationToken) [0x00146] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest+AsyncWrapper.GetResponseOrStream () [0x0004d] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest.GetResponse () [0x00044] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest.GetResponseStream () [0x00000] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Backend.Jottacloud+<>c__DisplayClass51_1.<ParallelGet>b__0 () [0x0004a] in <9506f4c8d1134272aa6ea119baff3a59>:0 
  at System.Threading.Tasks.Task`1[TResult].InnerInvoke () [0x0000f] in <d13c8b563008422a8c5aaec0a74089cc>:0 
  at System.Threading.Tasks.Task.Execute () [0x00000] in <d13c8b563008422a8c5aaec0a74089cc>:0 <---

2022-01-09 03:22:37 +00 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Failed: duplicati-b0ecf865843e2496780512880cabed88d.dblock.zip.aes (481.67 MB)
2022-01-09 03:22:37 +00 - [Error-Duplicati.Library.Main.Operation.TestHandler-FailedToProcessFile]: Failed to process file duplicati-b0ecf865843e2496780512880cabed88d.dblock.zip.aes
System.AggregateException: One or more errors occurred. (The remote server returned an error: (404) Not Found.) ---> System.Net.WebException: The remote server returned an error: (404) Not Found.
  at System.Net.HttpWebRequest.GetResponseFromData (System.Net.WebResponseStream stream, System.Threading.CancellationToken cancellationToken) [0x00146] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest+AsyncWrapper.GetResponseOrStream () [0x0004d] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest.GetResponse () [0x00044] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest.GetResponseStream () [0x00000] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Backend.Jottacloud+<>c__DisplayClass51_1.<ParallelGet>b__0 () [0x0004a] in <9506f4c8d1134272aa6ea119baff3a59>:0 
  at System.Threading.Tasks.Task`1[TResult].InnerInvoke () [0x0000f] in <d13c8b563008422a8c5aaec0a74089cc>:0 
  at System.Threading.Tasks.Task.Execute () [0x00000] in <d13c8b563008422a8c5aaec0a74089cc>:0 
   --- End of inner exception stack trace ---
  at Duplicati.Library.Main.BackendManager.GetForTesting (System.String remotename, System.Int64 size, System.String hash) [0x00065] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Operation.TestHandler.DoRun (System.Int64 samples, Duplicati.Library.Main.Database.LocalTestDatabase db, Duplicati.Library.Main.BackendManager backend) [0x0042f] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
---> (Inner Exception #0) System.Net.WebException: The remote server returned an error: (404) Not Found.
  at System.Net.HttpWebRequest.GetResponseFromData (System.Net.WebResponseStream stream, System.Threading.CancellationToken cancellationToken) [0x00146] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <6bc04dcac0a443ee834a449c98b8ed9d>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest+AsyncWrapper.GetResponseOrStream () [0x0004d] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest.GetResponse () [0x00044] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Utility.AsyncHttpRequest.GetResponseStream () [0x00000] in <2a3ee711c7c04f6c957360f2cf183a7f>:0 
  at Duplicati.Library.Backend.Jottacloud+<>c__DisplayClass51_1.<ParallelGet>b__0 () [0x0004a] in <9506f4c8d1134272aa6ea119baff3a59>:0 
  at System.Threading.Tasks.Task`1[TResult].InnerInvoke () [0x0000f] in <d13c8b563008422a8c5aaec0a74089cc>:0 
  at System.Threading.Tasks.Task.Execute () [0x00000] in <d13c8b563008422a8c5aaec0a74089cc>:0 <---

The only one I can find in your log is duplicati-b0ecf865843e2496780512880cabed88d.dblock.zip.aes.
Were there others next to it that had the same fate? Farther out, what was Duplicati doing at the time?
The status bar would show this, and the log probably would too (if more of the surrounding was given).

Seen how? If in a Jottacloud UI, can Jottacloud get them? If they’re visible but not downloadable, maybe:

Do you generally run such a log? If so, look through it for any previous mentions of the now-missing file.

Browsing in database or a database bug report is possible but less reliable than an independent log file.

Did you restore something that was affected by the missing file, and also disable usage of local blocks?

no-local-blocks

--no-local-blocks = false
Duplicati will attempt to use data from source files to minimize the amount of downloaded data. Use this option to skip this optimization and only use remote data.

The AFFECTED command

Returns a report explaining what backup sets and files are affected by a remote file.

Using the Command line tools from within the Graphical User Interface

EDIT:

Duplicati checks the remote file list before and after backup. If something is missing, it logs a complaint.
Although you might have other ways to look at the remote file date, if need be you can list using below.

Duplicati.CommandLine.BackendTool.exe

Give it the URL from Export As Command-line and see if it says that your unobtainable files are present.
What date do they have? Although context is missing (see above), file dates might give a clue for issue.
Names present in the list can also be tested with a get to see whether they download or give the 404.

Correct, I only added the log for 1 file as for what I could see the issue was the same for the other ones. The files that caused problems other than duplicati-b0ecf865843e2496780512880cabed88d.dblock.zip.aes are duplicati-b8c3827e3e1d84237b37bbd993fdd4cff.dblock.zip.aes, duplicati-ba321a0ca8b7a4c28bae88551ef1f74ea.dblock.zip.aes and duplicati-b2b64e24921cd41cda67ba61030bd9e42.dblock.zip.aes . I have full logs for this weekly backup since 3 october, the log is in total 10143 lines… The logs for 2/01/2022 (no issues except some retries, which seem to happen quite often) and for 9/01/2022 are uploaded, see zip underneath. You can also see that duplicati-b0ecf865843e2496780512880cabed88d.dblock.zip.aes and duplicati-b8c3827e3e1d84237b37bbd993fdd4cff.dblock.zip.aes have been verified 2/1/2022 without a problem.
duplicati_jottacloud_s2_20220102-09.log.zip (21.4 KB)

Other than the log, I cannot tell you what was happening exactly (except verification) as I was not watching the interface at that time.

In the Jottacloud web UI. I was also able to do a test restore of the files affected by them (found out via the AFFECTED command). Just do make sure I tried to download duplicati-b0ecf865843e2496780512880cabed88d.dblock.zip.aes which worked nicely.

Yes, I started doing that after my initial problem reports, see further up in this topic. Files were mentioned earlier on without problems.

I did more old-school and moved the original files before starting the restore. The “no local blocks” option is something to remember when doing a test restore, big time saver… :wink:

Big thanks for your efforts!

and the other two got their first log on Jan 9 with a problem, which is odd because the backup doesn’t seem to be uploading any files. Is this expected? It might be, if the source area can go for weeks without a change.

The other thing that is odd here is that it looks like the backup starts with an immense number of downloads, going through dlist files, then dindex, then dblock (which is when some downloads are getting the 404 error).

This pattern is what a database recreate (or repair with missing database) would do, with dblock downloads meaning it’s looking for missing data – but usually that’s not found, and some more errors are logged at end.

auto-cleanup controls whether repair is attempted. Is it possible that you have it on, causing the downloads?

Here are your backup starts and ends:

2022-01-02 03:17:22 +00 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started
2022-01-02 03:17:23 +00 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  ()
2022-01-02 03:17:24 +00 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed:  (2.13 KB)
2022-01-02 03:18:11 +00 - [Information-Duplicati.Library.Main.Operation.DeleteHandler-DeleteResults]: No remote filesets were deleted
2022-01-02 03:18:11 +00 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  ()
2022-01-02 03:18:13 +00 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed:  (2.13 KB)
2022-01-02 03:18:13 +00 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20210919T031601Z.dlist.zip.aes (433.29 KB)
2022-01-02 03:18:14 +00 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-20210919T031601Z.dlist.zip.aes (433.29 KB)
...
2022-01-02 06:55:31 +00 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-b081844fa937e431ab9923b35d0e5291b.dblock.zip.aes (478.63 MB)
2022-01-02 06:57:03 +00 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-b081844fa937e431ab9923b35d0e5291b.dblock.zip.aes (478.63 MB)
2022-01-02 06:57:41 +00 - [Information-Duplicati.Library.Modules.Builtin.SendMail-SendMailComplete]: Email sent successfully using server: smtps://mail.gmx.com:465
2022-01-09 03:17:51 +00 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started
2022-01-09 03:17:52 +00 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  ()
2022-01-09 03:17:53 +00 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed:  (2.13 KB)
2022-01-09 03:18:37 +00 - [Information-Duplicati.Library.Main.Operation.DeleteHandler-DeleteResults]: No remote filesets were deleted
2022-01-09 03:18:37 +00 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  ()
2022-01-09 03:18:38 +00 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed:  (2.13 KB)
2022-01-09 03:18:39 +00 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20211128T032745Z.dlist.zip.aes (433.43 KB)
2022-01-09 03:18:40 +00 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-20211128T032745Z.dlist.zip.aes (433.43 KB)
...
2022-01-09 06:38:14 +00 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-b691c311ad8024faab658a81456b79c6c.dblock.zip.aes (491.09 MB)
2022-01-09 06:39:55 +00 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-b691c311ad8024faab658a81456b79c6c.dblock.zip.aes (491.09 MB)
2022-01-09 06:40:34 +00 - [Information-Duplicati.Library.Modules.Builtin.SendMail-SendMailComplete]: Email sent successfully using server: smtps://mail.gmx.com:465

I edited manual testing approaches into my last response, but I don’t know if you saw them before posting.
The TEST command for all files would be another I wouldn’t usually give, but you might already be doing download of the entire backup (for unknown reason) each time you backup, so it might be no more severe.

Playing with both individual file downloads and mass downloads might find some pattern to the 404 errors.

For the database, you can check your advanced options and also the Database screen, and maybe push Verify files button to see if it looks like you have a working database or somehow there’s none there…
Restore and many other things would also look broken. I think only backup can has the auto repair option.

Hi, well, a lot to verify here :wink: Let’s have a look.

That can be explainable. I only sent you the last 2 weeks of logging whereas the backup job exists already longer than that and is weekly executed. It also existed already before I enabled the comprehensive logging I got working now. As for the 2 missing files, they were verified without problems as well: ba321a0ca8b7a4c28bae88551ef1f74ea on 10 October and 12 December, b2b64e24921cd41cda67ba61030bd9e42 on 31 October and 21 November.
Furthermore, the source data is very stable. The only reason I am taking these backups is that I want to ensure the backups are valid.

Not sure if you are surprised by the order of the file downloads or the amount of files it downloads. I deliberately take a big sample - backup test percentage is 5% (but as I pointed out in another topic before, Duplicati doubles that percentage to 10%). Also, the backup data is ± 510 GB and the blocksize is 500MB. So we are talking about a sizeable backup here.

No, it is off.

I did the commandline LIST but with some strange results… If I run LIST without a file mentioned I get this result:

root@duplicati-duplicati12:/opt/duplicati# mono Duplicati.CommandLine.exe list "jottacloud://(SNIP)" 

Enter encryption passphrase: 
  Listing remote folder ...
  Downloading file (4.12 KB) ...
  Downloading file (433.43 KB) ...
  Downloading file (433.29 KB) ...
  Downloading file (222.79 KB) ...
  Downloading file (221.84 KB) ...
Listing filesets:
0	: 01/10/2022 15:38:00
1	: 11/28/2021 03:27:45
2	: 09/19/2021 03:16:01
3	: 07/18/2021 03:16:11
4	: 05/14/2021 13:08:07

When I give one of the “erroneous” files as a parameter I get this error:

root@duplicati-duplicati12:/opt/duplicati# mono Duplicati.CommandLine.exe list jottacloud://(SNIP) duplicati-b0ecf865843e2496780512880cabed88d.dblock.zip.aes

Enter encryption passphrase: 
  Listing remote folder ...
  Downloading file (4.12 KB) ...
  Downloading file (433.43 KB) ...
  Downloading file (433.29 KB) ...
  Downloading file (222.79 KB) ...
  Downloading file (221.84 KB) ...
  Downloading file (4.12 KB) ...

Duplicati.Library.Main.Volumes.InvalidManifestException: Invalid manifest detected, the field Blocksize has value 26214400 but the value 102400 was expected
  at Duplicati.Library.Main.Volumes.VolumeBase+ManifestData.VerifyManifest (System.String manifest, System.Int64 blocksize, System.String blockhash, System.String filehash) [0x0007e] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Volumes.VolumeReaderBase.ReadManifests (Duplicati.Library.Main.Options options) [0x0004c] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Volumes.VolumeReaderBase..ctor (System.String compressor, System.String file, Duplicati.Library.Main.Options options) [0x00021] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Volumes.FilesetVolumeReader..ctor (System.String compressor, System.String file, Duplicati.Library.Main.Options options) [0x00000] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Operation.ListFilesHandler.Run (System.Collections.Generic.IEnumerable`1[T] filterstrings, Duplicati.Library.Utility.IFilter compositefilter) [0x003a7] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Controller+<>c__DisplayClass21_0.<List>b__0 (Duplicati.Library.Main.ListResults result) [0x0001c] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String[]& paths, Duplicati.Library.Utility.IFilter& filter, System.Action`1[T] method) [0x0026f] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Controller.RunAction[T] (T result, Duplicati.Library.Utility.IFilter& filter, System.Action`1[T] method) [0x00007] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.Library.Main.Controller.List (System.Collections.Generic.IEnumerable`1[T] filterstrings, Duplicati.Library.Utility.IFilter filter) [0x00021] in <e60bc008dd1b454d861cfacbdd3760b9>:0 
  at Duplicati.CommandLine.Commands.List (System.IO.TextWriter outwriter, System.Action`1[T] setup, System.Collections.Generic.List`1[T] args, System.Collections.Generic.Dictionary`2[TKey,TValue] options, Duplicati.Library.Utility.IFilter filter) [0x00202] in <eb902dee3a2c4c3fbc8192580b4a42b2>:0 
  at (wrapper delegate-invoke) System.Func`6[System.IO.TextWriter,System.Action`1[Duplicati.Library.Main.Controller],System.Collections.Generic.List`1[System.String],System.Collections.Generic.Dictionary`2[System.String,System.String],Duplicati.Library.Utility.IFilter,System.Int32].invoke_TResult_T1_T2_T3_T4_T5(System.IO.TextWriter,System.Action`1<Duplicati.Library.Main.Controller>,System.Collections.Generic.List`1<string>,System.Collections.Generic.Dictionary`2<string, string>,Duplicati.Library.Utility.IFilter)
  at Duplicati.CommandLine.Program.ParseCommandLine (System.IO.TextWriter outwriter, System.Action`1[T] setup, System.Boolean& verboseErrors, System.String[] args) [0x00342] in <eb902dee3a2c4c3fbc8192580b4a42b2>:0 
  at Duplicati.CommandLine.Program.RunCommandLine (System.IO.TextWriter outwriter, System.IO.TextWriter errwriter, System.Action`1[T] setup, System.String[] args) [0x00002] in <eb902dee3a2c4c3fbc8192580b4a42b2>:0 
root@duplicati-duplicati12:/opt/duplicati#

I get similar errors for the other files that gave errors - but also for a file I did not have a problem with.
You also mention a “GET”, which I tried, see here:

root@duplicati-duplicati12:/opt/duplicati# mono Duplicati.CommandLine.exe GET jottacloud://(SNIP) duplicati-b0ecf865843e2496780512880cabed88d.dblock.zip.aes
Command not supported: GET
root@duplicati-duplicati12:/opt/duplicati#

Now either I misread the documentation, or there is really an issue with my installation of Duplicati commandline. FYI, I am using Duplicati Duplicati - 2.0.6.3_beta_2021-06-17 on Docker (running on a Synology DS216-II+).

So, in short, we can conclude that I was not able to verify the existance of the files on the Jottacloud storage (other than using their GUI client or doing a test restore) and that there are some strange issues going on with the commandline…

Main problem may be using the wrong program. Extra Downloading from list might be explainable here.
Don’t run Duplicati.CommandLine.exe. Run below:

Ok… this was just an embarrassingly stupid mistake :wink:

Well, I did run the backendtool this time, which worked better :wink: But the result is that the files really seem to be there. When doing the LIST the files were found, here are the timestamps:

Found filenames Last change
duplicati-b0ecf865843e2496780512880cabed88d.dblock.zip.aes 05-03-21
duplicati-b8c3827e3e1d84237b37bbd993fdd4cff.dblock.zip.aes 05-02-21
duplicati-ba321a0ca8b7a4c28bae88551ef1f74ea.dblock.zip.aes 05-03-21
duplicati-b2b64e24921cd41cda67ba61030bd9e42.dblock.zip.aes 04/30/2021 10:59:18

After that I ran the GET commands on all the files:


root@duplicati-duplicati12:/opt/duplicati# mono Duplicati.CommandLine.BackendTool.exe GET "jottacloud://(SNIP)" duplicati-b0ecf865843e2496780512880cabed88d.dblock.zip.aes
root@duplicati-duplicati12:/opt/duplicati# mono Duplicati.CommandLine.BackendTool.exe GET "jottacloud://(SNIP)" duplicati-b8c3827e3e1d84237b37bbd993fdd4cff.dblock.zip.aes
root@duplicati-duplicati12:/opt/duplicati# mono Duplicati.CommandLine.BackendTool.exe GET "jottacloud://(SNIP)" duplicati-ba321a0ca8b7a4c28bae88551ef1f74ea.dblock.zip.aes
root@duplicati-duplicati12:/opt/duplicati# mono Duplicati.CommandLine.BackendTool.exe GET "jottacloud://(SNIP)" duplicati-b2b64e24921cd41cda67ba61030bd9e42.dblock.zip.aes
root@duplicati-duplicati12:/opt/duplicati# ls -l *.dblock.zip.aes
-rw-r--r-- 1 root root 505068685 Jan 13 20:00 duplicati-b0ecf865843e2496780512880cabed88d.dblock.zip.aes
-rw-r--r-- 1 root root 505962381 Jan 13 20:07 duplicati-b2b64e24921cd41cda67ba61030bd9e42.dblock.zip.aes
-rw-r--r-- 1 root root 522419533 Jan 13 20:03 duplicati-b8c3827e3e1d84237b37bbd993fdd4cff.dblock.zip.aes
-rw-r--r-- 1 root root 519962541 Jan 13 20:04 duplicati-ba321a0ca8b7a4c28bae88551ef1f74ea.dblock.zip.aes
-rw-r--r-- 1 root root         0 Aug 23 13:04 file.dblock.zip.aes

The empty file “file.dblock.zip.aes” is an unrelated file that remained on the location. And the modify dates in the list above are in line with the download time.

So the files seem to be there… and the modify dates seem to make sense, I did the first (bulk) backup during that period…

Is that edited or re-captioned? My list command looks quite different but does have a Size column.
I’m losing track a little, but the original post had a size issue, then later there were “(404) Not Found.”

I’m glad they’re there now. I wonder if something is intermittent, being upset about the data rate, etc?
I think the dblock files are more prone to this, right? If so, what’s different I wonder, besides the size?

It is edited. The LIST command did give back sizes and modification dates, but it returned all the files of the backup and I had no (obvious) way of filtering out the files that gave the problems. I edited it Excel (Calc, to be precise).

The original post, dating from August, was because of (apparent…) issues with the size of the DBLOCK files. You asked me to put on extensive logging, which I did. For some reason (I might have increased the retry count - other than that, no idea), I didn’t have any (blocking - I’ll elaborate further on that) problems after that, until last Sunday where I had these 404 errors. I reopened this forum because I thought it would give some context, that’s all :wink:

As for the size issue: I did have a look at the logs on a regular basis, and I remember it happened some times before, but then Duplicati would try to redownload the file - which solved everything. This happened on a weekly basis (not sure if it was always for size issues) but always solved after a retry. This is what I mean with “blocking” above - if Duplicati solves the issues itself by retrying the download, it doesn’t interfere with the verification process.

I’m believing more and more that it is a temporary glitch on Jottacloud site. The backup job will run again next Sunday, we’ll see what happens. If in the past I had errors, it was more often the DBLOCK ones than something else. But they are by far the biggest, so if anything goes wrong in storage or during transit, they have the biggest chance of being impacted. I didn’t give it much thought.

1 Like

Hi, Job run again today with no errors. A few retries, but this is something I got accustomed to, I guess.
For the moment I am assuming a Jottacloud glitch. I’ll keep an eye on it.
Thanks for your help!

1 Like