Missing files on remote

[jberry02]

Windows 10 desktop, still getting this error:

Found 4 files that are missing from the remote storage, please run repair

Clicking on the “show” button brings up the list of logs, but there is no log for the error.

Watching the live log (verbose) while attempting to run the backup doesn’t show anything of note other than the filenames of the missing files.

Clicking on “Repair” from the error pop-up briefly shows “Starting” but nothing else. Watching the live log shows:

Sep 26, 2025 11:49 AM: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (The CancellationTokenSource has been disposed.)
System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (The CancellationTokenSource has been disposed.)

 ---> System.ObjectDisposedException: The CancellationTokenSource has been disposed.

   at Duplicati.Server.Runner.<>c__DisplayClass16_2.<<RunInternal>b__3>d.MoveNext()

   --- End of inner exception stack trace ---
Sep 26, 2025 11:49 AM: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (The CancellationTokenSource has been disposed.)
System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (The CancellationTokenSource has been disposed.)

 ---> System.ObjectDisposedException: The CancellationTokenSource has been disposed.

   at Duplicati.Server.Runner.<>c__DisplayClass16_2.<<RunInternal>b__3>d.MoveNext()

   --- End of inner exception stack trace ---
Sep 26, 2025 11:49 AM: Backend event: List - Completed: (647 bytes) 

Running repair using the Commandline link shows:

Running commandline entry
Finished!

            
  Listing remote folder ...
The operation Repair has failed => The backup storage destination is missing data files. You can either enable `--rebuild-missing-dblock-files` or run the purge-broken-files command to remove these files. The following files are missing: duplicati-b5c6017a09aaf4200b1f3af2a41046fcf.dblock.zip.aes, duplicati-b69957746261648a185cf92b1edf76a28.dblock.zip.aes


ErrorID: MissingDblockFiles
The backup storage destination is missing data files. You can either enable `--rebuild-missing-dblock-files` or run the purge-broken-files command to remove these files. The following files are missing: duplicati-b5c6017a09aaf4200b1f3af2a41046fcf.dblock.zip.aes, duplicati-b69957746261648a185cf92b1edf76a28.dblock.zip.aes
Return code: 100

Adding –rebuild-missing-dblock-files as suggested results in:

Running commandline entry
Finished!

            
  Listing remote folder ...
  Uploading file duplicati-b4942c33d4a0747bab3006ccce6772313.dblock.zip.aes (49.31 MiB) ...
  Uploading file duplicati-i7b0dd0e313544dcda224202912dbf06f.dindex.zip.aes (3.97 KiB) ...
  Deleting file duplicati-b5c6017a09aaf4200b1f3af2a41046fcf.dblock.zip.aes  (49.31 MiB) ...
  Deleting file duplicati-i458ad3a4616245f5b5a7d1c39b653edf.dindex.zip.aes  (3.98 KiB) ...
  Uploading file duplicati-ba9bf87819f6d4d65a2b0f03c23d9e163.dblock.zip.aes (41.76 MiB) ...
  Uploading file duplicati-id890b4175116462eada4543f6d78666e.dindex.zip.aes (94.06 KiB) ...
  Deleting file duplicati-b69957746261648a185cf92b1edf76a28.dblock.zip.aes  (41.76 MiB) ...
  Deleting file duplicati-i512f3038f36e447792ea3f7ca85759d4.dindex.zip.aes  (94.06 KiB) ...
  Uploading file duplicati-if95af7613d7d4ea5bc2832da031c7fcd.dindex.zip.aes (5.40 KiB) ...
Return code: 0

At this point I can run the backup.

[jberry02]

Fedora 42 also still getting:

Found 1 files that are missing from the remote storage, please run repair

On this system for this instance the repair worked as expected and a subsequent backup ran without apparent error.

[kenkendk]

I moved this from the release post as I think these are not related to the release, but are more general.

@jberry02

Found 4 files that are missing from the remote storage, please run repair

Any idea why the files are suddenly missing? Did anything happen prior to this that could point to why the files would suddenly be (considered) missing? What backend is this?

@jberry02

At this point I can run the backup.

Great! So the issue is solved, but we do not know why the issue popped up in the first place?

@jberry02

Fedora 42 also still getting:

Found 1 files that are missing from the remote storage, please run repair

Are you getting this regularly?

[jberry02]

@kenkendk

I moved this from the release post as I think these are not related to the release, but are more general.

Good idea since this has been an on-going issue over several releases now.

@kenkendk

@jberry02

Found 4 files that are missing from the remote storage, please run repair

Any idea why the files are suddenly missing? Did anything happen prior to this that could point to why the files would suddenly be (considered) missing? What backend is this?

When I originally reported this with .100 the issue as near as I could tell appeared to be a 503 status from b2 that was being treated as success. I can post fresh log entries the next time it happens if desired/needed.

@kenkendk

Are you getting this regularly?

Yes. Typically 1 or 2 every day, usually on the job that backs up the Duplicati databases. It is quite annoying.

[jberry02]

Laptop #1, Fedora 42, b2 backend, 1/4 jobs failed with “Found 3 files that are missing from the remote storage, please run repair”

• repair fails with “The backup storage destination is missing data files. You can either enable `–rebuild-missing-dblock-files` or run the purge-broken-files command to remove these files. The following files are missing: duplicati-bbef36f0cae2f472f823426b413c6d195.dblock.zip.aes”
• “purge-broken-files” worked and a subsequent backup ran without reporting any errors.

Laptop #2, Fedora 42, b2 backend, 1/4 jobs failed with “Found 2 files that are missing from the remote storage, please run repair”

• clicking ”Repair” (ngax UI) shows no error, but a subsequent backup still reports the missing files. “Show log” does not show a log for the repair. Repair from commandline reports:

Running commandline entry
Finished!

            
  Listing remote folder ...
The operation Repair has failed => The backup storage destination is missing data files. You can either enable `--rebuild-missing-dblock-files` or run the purge-broken-files command to remove these files. The following files are missing: duplicati-b8a41e76d56d84b6cba70f41eb4f3afca.dblock.zip.aes

ErrorID: MissingDblockFiles
The backup storage destination is missing data files. You can either enable `--rebuild-missing-dblock-files` or run the purge-broken-files command to remove these files. The following files are missing: duplicati-b8a41e76d56d84b6cba70f41eb4f3afca.dblock.zip.aes
Return code: 100

• Repair with –rebuild-missing-dblock-files reports:

Running commandline entry
Finished!

            
  Listing remote folder ...
Repair acquired 105 blocks for volume duplicati-b8a41e76d56d84b6cba70f41eb4f3afca.dblock.zip.aes, but 8 blocks are still missing. If you want to continue working with the database, you can use the "list-broken-files" and "purge-broken-files" commands to purge the missing data from the database and the remote storage.
  Uploading file duplicati-bca2dcb2662da48d78c8029122185c54d.dblock.zip.aes (48.264 MiB) ...
  Uploading file duplicati-i0c5e58e2f6734f769d6a7ff225488af3.dindex.zip.aes (5.263 KiB) ...
  Uploading file duplicati-i0ae8ba7f9267411da04ccbb6ab7f14f9.dindex.zip.aes (5.544 KiB) ...
Return code: 0

• purge-broken-files reports:

Running commandline entry
Finished!

            
  Listing remote folder ...
  Uploading file duplicati-20250929T073001Z.dlist.zip.aes (1.888 KiB) ...
  Deleting file duplicati-20250929T073000Z.dlist.zip.aes  ...
Return code: 0

• a subsequent backup runs without apparent errors

I also have the same failure on 1/3 backups on a Windows 10 desktop, but don’t have the time to marshall the data at the moment - will report back later.

I have log files for all of these failures that span the prior backup (when the failure actually occurred) as well as this morning’s backup and can provide them if desired.

[jberry02]

@jberry02

I also have the same failure on 1/3 backups on a Windows 10 desktop, but don’t have the time to marshall the data at the moment - will report back later.

Windows 10, b2, “Found 3 files that are missing from the remote storage, please run repair“

• Clicking Repair (ngax UI) completes almost instantly with no error shown, attempting to run backup re-raises the missing files error.
• “Show log” from the backup does not show any log for the Repair
• “About” → “Show log” has the log from the repair, and shows that it failed:

 Sep 29, 2025 10:05 AM: Failed while executing Repair "delli2-Duplicati-Config-Dir" (id: 7)
Duplicati.Library.Interface.UserInformationException: The backup storage destination is missing data files. You can either enable `--rebuild-missing-dblock-files` or run the purge-broken-files command to remove these files. The following files are missing: duplicati-b1fc049de20724fee8a75c5e1b0f906df.dblock.zip.aes

• purge-broken-files gives:

Running commandline entry
Finished!

            
  Listing remote folder ...
  Uploading file duplicati-20250929T074001Z.dlist.zip.aes (2.50 KiB) ...
  Deleting file duplicati-20250929T074000Z.dlist.zip.aes  ...
Return code: 0

• now the backup runs without any apparent errors

[jberry02]

No “missing files” errors reported on my .103 systems today.

[jberry02]

Fedora 42 system #1, b2 backend - 1/4 backups failed with 4 missing files.

Fedora 42 system #2, b2 backend - 1/4 backups failed with 2 missing files.

[jberry02]

Today, same systems, same errors including the number of files. I don’t know that I have seen such a coincidence before.

Is there value in my providing a daily report on these errors?

[jberry02]

Fedora 42 system #1, b2 backend - 1/4 backups failed with 11 (eleven) missing files.

[jberry02]

Windows 10, b2 backend, 1/3 backups failed with 8 missing files.

[jberry02]

Fedora 42 system #1, b2 backend - 2/4 backups failed: 6 missing files, 5 missing files.

Windows 10, b2 backend, 1/3 backups failed with 1 missing file.

[jberry02]

Fedora 42 system #1, b2 backend - 1/4 backups failed with 3 missing files.

Windows 10, b2 backend, 1/3 backups failed with 1 missing file.

Fedora 42 system #2, b2 backend - 1/4 backups failed with 1 missing file.

I will probably update to .104 later today, but I don’t see anything in the “release” post that suggests this issue has been addressed.

[jberry02]

(I have not yet updated to .104 - this is still from .103)

Fedora 42 system #1, b2 backend - 1/4 backups failed with 3 missing files.

[ts678]

@jberry02

When I originally reported this with .100 the issue as near as I could tell appeared to be a 503 status from b2 that was being treated as success.

Below is likely reference. Discussion followed by some, but I don’t think dev commented.

2025-08-15 03:41:46 -04 - [Verbose-Duplicati.Library.Main.NetworkTrafficLogger.Http-ResponseHeadersStop]: [System.Net.Http] Event: ResponseHeadersStop
  statusCode          : 503

Backblaze B2 uploads fail, often with timeout. Big read-write-timeout may help.

was said (by GitHub version note) to be fixed in 2.1.1.100 by

Removing chunksize from B2 backend #6441

and yet it seems like it was seen in 2.1.1.100 in quote above, along with the old note:

@jberry02

I can post fresh log entries the next time it happens if desired/needed.

I, for one, would be interested in looking deeper. There are also ideas at:

2.1.1.104 > Files are missing from the remote storage

One unanswered question is whether missing files got a statusCode 200.

Since the previous problem is read-write-timeout, does increasing it help?

If there is an unusual remote volume size or advanced options, mention it.

Running BackendTester would be good. In early worse days, that got odd.

[jberry02]

@ts678

2.1.1.104 > Files are missing from the remote storage

This appears to be the same issue. Same (or at least very similar) results from repair attempts, etc.

@ts678

Since the previous problem is read-write-timeout, does increasing it help?

read-write-timeout helped with a prior issue that also resulted in missing files. I have not noticed it helping with this error. B2 responds - hence no timeout - but it appears duplicati treats the response as “success” when it is actually a failure.

If there is an unusual remote volume size or advanced options, mention it.

Everything on the duplicati side is default (other than where I would have increased the read-write-timeout, but I believe I reset that once I determined it wasn’t in-play). Everything with the b2 bucket is default except that I disabled versioning.

@ts678

@jberry02

I can post fresh log entries the next time it happens if desired/needed.

I, for one, would be interested in looking deeper.

DM me your e-mail; I’ll put a current copy of one of the logs on google drive.

[ts678]

Looking at what I guess are the 8 missing files you posted about on Oct 4, each upload
declared Completed shows statusCode 503 as prior non-blank line at the same second.

I think concurrent uploads can muddy line-matching, but tight timing increases credibility.

grep -E -B 3 'Put - Completed: duplicati-(<random-b-or-i-id alternation>)'

There are a few other odd things going on, such as statusCode of -1. They get retry, e.g

2025-10-05 03:44:37 -04 - [Verbose-Duplicati.Library.Main.NetworkTrafficLogger.Http-RequestStop]: [System.Net.Http] Event: RequestStop
  statusCode          : -1

2025-10-05 03:44:37 -04 - [Retry-Duplicati.Library.Main.Backend.Handler-RetryPut]: Operation Put with file duplicati-ba4bf0be5a2c44ec1bcef1af6096b635e.dblock.zip.aes attempt 2 of 5 failed with message: An error occurred while sending the request.
System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.Net.Http.HttpIOException: The response ended prematurely. (ResponseEnded)
   at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.<SendCoreAsync>g__Core|4_0(HttpRequestMessage request, Boolean useAsync, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.<SendCoreAsync>g__Core|4_0(HttpRequestMessage request, Boolean useAsync, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Duplicati.Library.JsonWebHelperHttpClient.GetResponseAsync(HttpRequestMessage req, HttpCompletionOption httpCompletionOption, CancellationToken cancellationToken)
   at Duplicati.Library.JsonWebHelperHttpClient.GetResponseAsync(HttpRequestMessage req, HttpCompletionOption httpCompletionOption, CancellationToken cancellationToken)
   at Duplicati.Library.JsonWebHelperHttpClient.ReadJsonResponseAsync[T](HttpRequestMessage req, CancellationToken cancellationToken)
   at Duplicati.Library.JsonWebHelperHttpClient.GetJsonDataAsync[T](String url, CancellationToken cancellationToken, Action`1 setup)
   at Duplicati.Library.JsonWebHelperHttpClient.PostAndGetJsonDataAsync[T](String url, Object item, CancellationToken cancellationToken)
   at Duplicati.Library.Backend.Backblaze.B2.<>c__DisplayClass30_0.<<RebuildFileCache>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Duplicati.Library.Utility.Utility.WithTimeout[T](TimeSpan timeout, CancellationToken token, Func`2 func)
   at Duplicati.Library.Backend.Backblaze.B2.RebuildFileCache(CancellationToken cancellationToken)
   at Duplicati.Library.Backend.Backblaze.B2.PutAsync(String remotename, Stream stream, CancellationToken cancelToken)
   at Duplicati.Library.Main.Backend.BackendManager.PutOperation.PerformUpload(IBackend backend, String hash, Int64 size, CancellationToken cancelToken)
   at Duplicati.Library.Main.Backend.BackendManager.PutOperation.ExecuteAsync(IBackend backend, CancellationToken cancelToken)
   at Duplicati.Library.Main.Backend.BackendManager.Handler.Execute[TResult](PendingOperation`1 op, CancellationToken cancellationToken)
   at Duplicati.Library.Main.Backend.BackendManager.Handler.Execute(PendingOperationBase op, CancellationToken cancellationToken)
   at Duplicati.Library.Main.Backend.BackendManager.Handler.ExecuteWithRetry(PendingOperationBase op, CancellationToken cancellationToken)
2025-10-05 03:44:37 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Retrying: duplicati-ba4bf0be5a2c44ec1bcef1af6096b635e.dblock.zip.aes ()

Once again, I’m relying on log proximity and same second timing to connect the lines.
Developer input would be nice on why statusCode 503 is considered Completed here.

EDIT 1:

To explain the grep, it shows three lines of context before Completed of missing file:

2025-10-04 03:40:44 -04 - [Verbose-Duplicati.Library.Main.NetworkTrafficLogger.Http-RequestStop]: [System.Net.Http] Event: RequestStop
  statusCode          : 503

2025-10-04 03:40:44 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-ba4e062e7f3624ee48468af8b4e89c79a.dblock.zip.aes (49.19 MiB)
--
2025-10-04 03:40:44 -04 - [Verbose-Duplicati.Library.Main.NetworkTrafficLogger.Http-RequestStop]: [System.Net.Http] Event: RequestStop
  statusCode          : 503

2025-10-04 03:40:44 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-i5eebbd2c0f1d42e29a8d25a735198552.dindex.zip.aes (7.45 KiB)
--
2025-10-04 03:41:18 -04 - [Verbose-Duplicati.Library.Main.NetworkTrafficLogger.Http-RequestStop]: [System.Net.Http] Event: RequestStop
  statusCode          : 503

2025-10-04 03:41:18 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-bb94e09f0e70c44cfabab968f7e301447.dblock.zip.aes (49.07 MiB)
--
2025-10-04 03:41:20 -04 - [Verbose-Duplicati.Library.Main.NetworkTrafficLogger.Http-RequestStop]: [System.Net.Http] Event: RequestStop
  statusCode          : 503

2025-10-04 03:41:20 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-ie7826635934b4c2b938778eb3d7de76c.dindex.zip.aes (3.97 KiB)
--
2025-10-04 03:41:46 -04 - [Verbose-Duplicati.Library.Main.NetworkTrafficLogger.Http-RequestStop]: [System.Net.Http] Event: RequestStop
  statusCode          : 503

2025-10-04 03:41:46 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b82b27c91957e45ebb2c8180bc31e535d.dblock.zip.aes (49.30 MiB)
--
2025-10-04 03:41:48 -04 - [Verbose-Duplicati.Library.Main.NetworkTrafficLogger.Http-RequestStop]: [System.Net.Http] Event: RequestStop
  statusCode          : 503

2025-10-04 03:41:48 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-i8865aa718cb84ed4ab419f06a28b1000.dindex.zip.aes (4.43 KiB)
--
2025-10-04 03:42:06 -04 - [Verbose-Duplicati.Library.Main.NetworkTrafficLogger.Http-RequestStop]: [System.Net.Http] Event: RequestStop
  statusCode          : 503

2025-10-04 03:42:06 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b0b49d0562b174cc79e0d2bb3409bba8b.dblock.zip.aes (49.41 MiB)
--
2025-10-04 03:42:07 -04 - [Verbose-Duplicati.Library.Main.NetworkTrafficLogger.Http-RequestStop]: [System.Net.Http] Event: RequestStop
  statusCode          : 503

2025-10-04 03:42:07 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-i95fc044b1bcc466aaca97ff87c61020c.dindex.zip.aes (5.81 KiB)

[jberry02]

@ts678

Developer input would be nice on why statusCode 503 is considered Completed here.

Your analysis matches what I posted a few versions back - upload receives 503, duplicati treats it as success, next backup fails verify because files are missing.

[jberry02]

No missing files reported today.

[jberry02]

Systems listed below have been upgraded to Duplicati - 2.1.1.104_canary_2025-10-03

Windows 10, b2 backend, 1/3 backups failed with 1 missing file.

Fedora 42 system #2, b2 backend - 1/4 backups failed with 1 missing file.