Backups broken on running job when destination disk is not available

Support
1

“Server” runs on SSD 512GB plus HDD 14TB.
Setup 3 backup jobs time daily scheduled to the same desination drive.
For energy consuption reasons the “Server” goes to Standby during night hours and is started in the Morning manually when needed. For unknown reasons the 14TB HDD fails to start when the Sytem is going up. This happend every some weeks within the last months.
I know this should not be, but its impossible to ask for replacement of the HDD even it is still under warranty since there are is nothing suspicious in smart data. On the other hand the system is quite old and to be replaced, so the reason might be a week capacitor.

Well today the system was started from PowerDown at 9:36am, I did not recocknize the HDD did not start poperly so Ducpikati started backups scheduled at 1:10,1:20,1:30pm
For sure all three backups failed but now two of it are missing files now and ask for repair.

The backup job that still works after the disk is available again showed this error message:

2025-04-28 13:10:46 +02 - [Error-Duplicati.Library.Main.Controller-FailedOperation]: Die Operation Backup ist mit folgenden Fehler fehlgeschlagen: Ein nicht vorhandenes Gerät wurde angegeben. : '\\?\h:\Duplicati-backup_local\Datenarchiv'
IOException: Ein nicht vorhandenes Gerät wurde angegeben. : '\\?\h:\Duplicati-backup_local\Datenarchiv'

One backup job that now needs repair:

2025-04-28 13:32:14 +02 - [Error-Duplicati.Library.Main.Operation.TestHandler-FailedToProcessFile]: Failed to process file duplicati-20250428T113000Z.dlist.zip.aes
IOException: Ein nicht vorhandenes Gerät wurde angegeben. : ‘\?\h:\Duplicati-backup_local\Privatdaten\duplicati-20250428T113000Z.dlist.zip.aes’
2025-04-28 13:32:55 +02 - [Error-Duplicati.Library.Main.Operation.TestHandler-FailedToProcessFile]: Failed to process file duplicati-20241210T082545Z.dlist.zip.aes
IOException: Ein nicht vorhandenes Gerät wurde angegeben. : ‘\?\h:\Duplicati-backup_local\Privatdaten\duplicati-20241210T082545Z.dlist.zip.aes’
2025-04-28 13:33:35 +02 - [Error-Duplicati.Library.Main.Operation.TestHandler-FailedToProcessFile]: Failed to process file duplicati-20250425T163459Z.dlist.zip.aes
IOException: Ein nicht vorhandenes Gerät wurde angegeben. : ‘\?\h:\Duplicati-backup_local\Privatdaten\duplicati-20250425T163459Z.dlist.zip.aes’
2025-04-28 13:34:16 +02 - [Error-Duplicati.Library.Main.Operation.TestHandler-FailedToProcessFile]: Failed to process file duplicati-i4322cee6c6ec4edda054f41aeda54c8e.dindex.zip.aes
IOException: Ein nicht vorhandenes Gerät wurde angegeben. : ‘\?\h:\Duplicati-backup_local\Privatdaten\duplicati-i4322cee6c6ec4edda054f41aeda54c8e.dindex.zip.aes’
2025-04-28 13:34:57 +02 - [Error-Duplicati.Library.Main.Operation.TestHandler-FailedToProcessFile]: Failed to process file duplicati-i5a8fd7a0d01f479fa34b8c7e18690483.dindex.zip.aes
IOException: Ein nicht vorhandenes Gerät wurde angegeben. : ‘\?\h:\Duplicati-backup_local\Privatdaten\duplicati-i5a8fd7a0d01f479fa34b8c7e18690483.dindex.zip.aes’
2025-04-28 13:35:38 +02 - [Error-Duplicati.Library.Main.Operation.TestHandler-FailedToProcessFile]: Failed to process file duplicati-if72e7cfaefc1490c9bc8ef4f2070f1d0.dindex.zip.aes
IOException: Ein nicht vorhandenes Gerät wurde angegeben. : ‘\?\h:\Duplicati-backup_local\Privatdaten\duplicati-if72e7cfaefc1490c9bc8ef4f2070f1d0.dindex.zip.aes’
2025-04-28 13:36:18 +02 - [Error-Duplicati.Library.Main.Operation.TestHandler-FailedToProcessFile]: Failed to process file duplicati-b084f571b936c408c9c87092f05ca12a2.dblock.zip.aes
IOException: Ein nicht vorhandenes Gerät wurde angegeben. : ‘\?\h:\Duplicati-backup_local\Privatdaten\duplicati-b084f571b936c408c9c87092f05ca12a2.dblock.zip.aes’
2025-04-28 13:36:59 +02 - [Error-Duplicati.Library.Main.Operation.TestHandler-FailedToProcessFile]: Failed to process file duplicati-b9dda665db1f54ecd8a7fd6b1c6753552.dblock.zip.aes
IOException: Ein nicht vorhandenes Gerät wurde angegeben. : ‘\?\h:\Duplicati-backup_local\Privatdaten\duplicati-b9dda665db1f54ecd8a7fd6b1c6753552.dblock.zip.aes’
2025-04-28 13:37:40 +02 - [Error-Duplicati.Library.Main.Operation.TestHandler-FailedToProcessFile]: Failed to process file duplicati-bcbc9fe8f41c44322b51cbb70dd67ed92.dblock.zip.aes
IOException: Ein nicht vorhandenes Gerät wurde angegeben. : ‘\?\h:\Duplicati-backup_local\Privatdaten\duplicati-bcbc9fe8f41c44322b51cbb70dd67ed92.dblock.zip.aes’
2025-04-28 13:37:40 +02 - [Error-Duplicati.Library.Main.Operation.TestHandler-Test results]: Verified 9 remote files with 9 problem(s)

And throws this started with attached destination disk:

Errors 1
* 2025-04-28 15:25:48 +02 - [Error-Duplicati.Library.Main.Controller-FailedOperation]: Die Operation Backup ist mit folgenden Fehler fehlgeschlagen: Found 3 files that are missing from the remote storage, please run repair RemoteListVerificationException: Found 3 files that are missing from the remote storage, please run repair

I had the problem once before and a repair worked to fix the problem. But how can be a missing destination disk for whatever reason is messing up the backup setup ? And why in 2 of 3 cases?
Please let me know i there is anything to save/provide before I run the repair.

2

I start loosing my faith in Duplicati …
Started repair - not working:

backup storage destination is missing data files. You can either enable --rebuild-missing-dblock-files or run the purge command to remove these files. The following files are missing: duplicati-bab78596455bc441cb742e7f77381fc21.dblock.zip.aes
UserInformationException: The backup storage destination is missing data files. You can either enable --rebuild-missing-dblock-files or run the purge command to remove these files. The following files are missing: duplicati-bab78596455bc441cb742e7f77381fc21.dblock.zip.aes

started purge:

  • 2025-04-29 10:11:11 +02 - [Error-Duplicati.Library.Main.Controller-FailedOperation]: Die Operation PurgeFiles ist mit folgenden Fehler fehlgeschlagen: Unable to start the purge process as there are 5 orphan file(s) UserInformationException: Unable to start the purge process as there are 5 orphan file(s)

started repair --rebuild-missing-dblock-files:

Listing remote folder …
Uploading file duplicati-20250428T113000Z.dlist.zip.aes (175,76 KB) …
Failed to perform cleanup for missing file: duplicati-b39acac422b0d4ac299b2eba219fb5a3b.dblock.zip.aes, message: Repair not possible, missing 4 blocks.
If you want to continue working with the database, you can use the “list-broken-files” and “purge-broken-files” commands to purge the missing data from the database and the remote storage. => Repair not possible, missing 4 blocks.
If you want to continue working with the database, you can use the “list-broken-files” and “purge-broken-files” commands to purge the missing data from the database and the remote storage.
Uploading file duplicati-id7042947f4c14a90a0906aaa875d81b8.dindex.zip.aes (1,04 KB) …
Return code: 0

i run purge-broken-files:

Fertiggestellt!

Listing remote folder …
Return code: 0
start the backup job again:
Warnings 2

  • 2025-04-29 10:29:16 +02 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingFile]: Missing file: duplicati-bab78596455bc441cb742e7f77381fc21.dblock.zip.aes
  • 2025-04-29 10:29:16 +02 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingFile]: Missing file: duplicati-i7c2a7699a1854d7db9c7f1c774059f93.dindex.zip.aes

Errors 1

  • 2025-04-29 10:29:16 +02 - [Error-Duplicati.Library.Main.Controller-FailedOperation]: Die Operation Backup ist mit folgenden Fehler fehlgeschlagen: Found 2 files that are missing from the remote storage, please run repair RemoteListVerificationException: Found 2 files that are missing from the remote storage, please run repair
3

The show goes on:
Database deletean rebuilt for one of the two spoilt backup jobs was successful. Backup jog succeeded

But doing the same procedure for the second DB even the rebulit did not work:

Error: 2025-04-29 12:17:33 +02 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-MissingFileDetected]: Remote file referenced as duplicati-b39acac422b0d4ac299b2eba219fb5a3b.dblock.zip.aes by duplicati-id7042947f4c14a90a0906aaa875d81b8.dindex.zip.aes, but not found in list, registering a missing remote file

executed another repair and finally I could run the backup job successfully. So all is fixed again.

The problem is, I have to run procedues I don’t really know what they do (and actually I don"t like to care about) in the hope not loosing my backup data - imho this not backups are intented to be made for.

4

That sounds as expected. “Device does not exist”, which is consistent with the disk not being available.

This is more of the same, but for some reason this process has managed to get past the “list remote folder” stage and is now testing each of the files (which it cannot access).

Are you running this as a separate test step or is it part of the backup? Usually, the testing part does not happen until the end of the backup, if you are running testing as part of the backup process.

In principle, if no files have changed, there is no need to access the remote destination, which could make the backup fly through the process, and end up in the Test step.

But, there is an initial “list-verify-remote” that will check that all files exist on the remote, which should have complained and failed the job. Do you have the option --no-backend-verification set?

The term “missing” here means that the database has recorded that the file was uploaded (or stored on H: in this case) without any errors. And now when listing the remote storage, the files are no longer there.

And you are quite right this should not happen in general, and should certainly not be possible if the remote destination is unavailable.

It is possible that the unavailability is unrelated, and the files have disappeared.

Could you create a bugreport database and upload it somewhere (Wetransfer, Dropbox, etc) and share a link with me in a PM ?

Screenshot 2025-03-19 at 16.16.03
Screenshot 2025-03-19 at 16.16.031102×746 50.7 KB

Hopefully with this database, I can trace the missing files and see where they originated from.

5

Do you have the option --no-backend-verification set? No I did not.
I forgot to mention this runs on a windows 10 machine, and I suspect the problem might be related to the fact windows is caching folder structures making it availbale even when the folder/device is not connected. So I can see the drive even if did not start after wake up. You have you dive into or try to open a file you detect the drive is gone.
I think you might be able to reproduce this by setting a win10 machine to Power Save and disconect the SATA cable of the destination drive before power up again. I created a bug report and uploaded it to my PCloud Drive. See PM.