Error message about remote volumes marked as deleted

[kenkendk]

@OverloadUT

Oh, and I can verify that the Fileset table has 70 rows while the Remotevolume table has 71 Files entries. (thanks for the tip on comparing those @ts678)

That is unexpected! So somehow there is a file that is known to Duplicati, but it is lacking a fileset.

I think the missing file hash is part of the explanation. Usually, the file is created locally and the hash is recorded in the database. Files without a hash are usually only present after a database recreate where the files have not yet been accessed. But since this is a dlist file it should always be downloaded during recreate.

If the file “magically” appears on the remote storage, there should be an error during the backup where it verifies that all known files are there and that no new files are suddenly existing. But in this case the status is Verified, meaning that it is known.

I can see that the the other timestamps are around 2025-02-xx, so this is an older file which may also be a hint that it has reappeared?

@OverloadUT

(last successful backup was 2024-11-07, and 2024-11-13 is the next timestamp we can see in the files)

It looks like there were backups running in 2025?

@OverloadUT

However, one of the records has an OperationID of 3, and its Hash is null.

That points to the Operation table. You should be able to see what operation was running as “operation 3”.

If you can create a bugreport database, I would love to look at it. If not, can you perhaps delete the tables with filenames (and run VACUUM after), and share a copy with me?
The remainder of the tables should not contain any sensitive information, but you can manually verify.

@OverloadUT

If this were me just throwing shit at the wall, I would try manually deleting the .aes file from the server and then removing that row from the Remotevolume table. But I’ve got no idea if that will blow everything up.

Pretty sure that would work. To be safe, rename the file instead of deleting it. If it starts with something other than duplicati- it will be ignored. Also make a copy of the database before removing the entry.

@OverloadUT

Or if you’d like me to preserve this state for root cause debugging!

I don’t think we will learn much more from the current state, but the logs in the database may help to trace down the cause.

@mathdu

here’s the bug report i could extract:

Thanks! That makes it a lot easier to track down.

@ts678

Below is from my first look,

Thanks for the summary and analysis.

My summary of your summary is:

• 433: regular backup
• 434: regular backup, but Fileset is missing, uploads seems silently failing
• 435: failed backup, never gets started as listing prior to verification failed
• 436: failed due to missing Fileset

It is possible that some later operation has deleted the Fileset from the database, but given the error is detected in 436, I don’t think this is the case.

From what I can see the problem seems to start with operation 434.
This operation completed without any errors and marked a remote volume as deleted.
I am assuming this operating also removed the Fileset that was associated with the remote dlist file that was deleted (duplicati-20250127T090316Z.dlist.zip).

But there is no fileset for operation 434 in the database. Not sure if this was deleted as part of the other delete or not, but my guess is that it was never created. There is a remote volume named duplicati-20250202T125444Z.dlist.zip that was created with version 434, and the stats say:

{
    "FilesUploaded": 1,
    "FilesDownloaded": 3,
    "FilesDeleted": 1
}

Which is suggests with 1 delete, 1 uploaded, and a triplet downloaded for testing.
However, examining the RemoteOperation table and the log shows that it did indeed upload a number of files. According to the RemoteOperation table there was a total of 17 files (attempted) uploaded.

It looks like none of these files were actually uploaded as they do not show up in RemoteVolumes or any subsequent listing of the destination. Also, all 17 files are dblock files, meaning the related dindex file is never uploaded.

This seems to indicate that it is possible for there to be a situation where uploads are started but never complete. I don’t know exactly how it can happen, but I can see that it is possible to close the channel that serves new work tasks. If this happens, the process shuts down without even looking at any in-progress tasks.

This could explain the missing uploads, and perhaps also the issue reported by @ts678.

I don’t understand how or why the channel is closed as it does not do this in any of our tests. I also don’t understand how the upload of the dlist can succeed while the dblock is either failed or in-progress.

@mathdu In the UI, do you see any errors for the backup that was running on 2025-02-02T13:12:00.9864685Z (UTC)? I am suspecting that something in the shutdown logic is perhaps causing errors that are not logged.

Fortunately, I have removed all the code in question here for 2.1.0.108, but it does not fix the setup.

Also, @mathdu do you want to try to recover the backup to the state before 2025-02-02 ?