Local/remote storage out-of-sync, but repair doesn't help

Support
1

Hi,

I’m running 2.1.0.108_canary_2025-01-31.

One of my backup jobs is in a weird state since a week or so. When I run the backup I see either:

  • Found 1 files that are missing from the remote storage, please run repair
  • Found 1 remote files that are not recorded in local storage, please run repair

When I run the repair job, I see this in the logs:

[Information-Duplicati.Library.Main.Operation.RepairHandler-DatabaseIsSynchronized]: Destination and database are synchronized, not making any changes

So, I’m not sure if it actually fixed anything or not.

After the repair run, the backup runs fine once. The second time it complains again about the missing file on one end or the other.

When I try to recreate the database via “Recreate (delete and repair)” nothing happens?!

What else can I do to get the backup back into a working state?

2

Hi @Horst, welcome to the forum.

What backend destination are you using?

This sounds like a delay of sorts, where the storage is slow in updating the listing.

How long apart are the runs? When it complains about a file that is missing, can you manually verify that the file exists on the remote destination?

I am guessing that this has do with some caching in a backend that is not behaving. With 2.1.0.8 I changed the way backends are (re-)used, and if the caching is not working correctly across the instances, this could explain the problem.

If this is indeed the problem, I am very interested in knowing what backend that is causing the problems.

The option --asynchronous-concurrent-upload-limit=1 can be set to avoid using multiple backend instances, and this should fix it, if it is indeed a cache issue.

3

The backend is mega.nz.

The files that are “missing” on the remote storage, are always there when I read the names from the logs, so, I guess, they are there just “late” for the verification step.

4

My first attempts with the option --asynchronous-concurrent-upload-limit=1 are promising. I haven’t seen any issues with this option yet.

5

From the next version of Duplicati, we will be giving a warning when using Mega.nz as they do not have a public API for use by non-Mega products and there is no maintained library for this anymore. Without a public API they could change the implementation at any moment and break the backend.

That confirms my guess then. I will try to make a common solution for this problem across backends.