Found XXX files that are missing from the remote storage

Support
1

Today, ~10 of my hosts failed to backup. When looking into the issue, one thread was common:

Found 80 files that are missing from the remote storage, please run repair

Yesterday, I found that 2 of my hosts were facing the same issue. So I ran --rebuild-missing-dblock-files, which didn’t work. Then I tried to PurgeBrokenFiles, which didn’t work. Then I tried to recreate the database, which didn’t work at first, and then it lost connection to the backend for some reason. I ended up deleting it entirely and starting again, but it was a few hours before it managed to connect to the backend and backup successfully.

And the very next day, the same issue happens, but for all of my hosts. So, I investigated further this time. I ran ListBrokenFiles for two hosts and got this error:

  Listing remote folder ...

ErrorID: CannotPurgeWithNoRemoteVolumes
No remote volumes were found, refusing purge
Return code: 100

I clicked “Test Connection” and that worked fine. So I went to the backend and found that the entire directory still exists, completely full of files. I ran Repair, which didn’t work, and told me this:

Duplicati.Library.Interface.UserInformationException: The backup storage destination is missing data files. You can either enable `--rebuild-missing-dblock-files` or run the purge command to remove these files. 
The following files are missing: 
duplicati-b64c541bf479f4c40ab4286a646b3f34f.dblock.zip.aes, 
duplicati-b94da1f6659aa47de919595acfcdcd291.dblock.zip.aes, 
duplicati-b4e553d09f9ac45818b4e4567fda9234d.dblock.zip.aes, 
duplicati-b6ea99285931f4c7caa49589da88322e9.dblock.zip.aes,
...

All these files are present in the remote storage and not zero’d out…

Any ideas? I’ve decided this is likely an issue connecting to the backend and not being able to read/write any data. I have no idea how to further debug this connection though.

Edit: I’m using OpenStack to connect to my OVH backend.

2

given the symptoms, I’d say that it very likely indeed.

If there is something misunderstood at the Json dialog level, Duplicati code should throw and it should be displayed in the UI (hopefully). If this is not the case, try to run a backup in a terminal (export the job as command line) to see if anything like a backtrace is displayed.

You are paying for the backend right ? How about asking them for some support ?

From browsing the Duplicati source code, there is not much more about Openstack than Http + json, so there is no specific driver that we can update, it’s all Duplicati code. All of the authors of this code have left Duplicati development, and there is AFAIK no one active having an Openstack setup. As far as I know there is no Openstack provider giving away a free no-string-attached account of a few GB for testing and eval purpose so I can’t test it myself.

3

Thanks for the response.

run backup in a terminal

I exported the job, but this is all I got:

Backup started at 07/13/2023 07:28:18
Checking remote backup ...
  Listing remote folder ...
Missing file: duplicati-b64c541bf479f4c40ab4286a646b3f34f.dblock.zip.aes
Missing file: duplicati-b94da1f6659aa47de919595acfcdcd291.dblock.zip.aes
...
Missing file: duplicati-ie5d7e510e5e04ecc92ac8198b68ce002.dindex.zip.aes

Found 80 files that are missing from the remote storage, please run repair
Fatal error => Found 80 files that are missing from the remote storage, please run repair

ErrorID: MissingRemoteFiles
Found 80 files that are missing from the remote storage, please run repair

You are paying for the backend right ? How about asking them for some support ?

I’ve submitted a support ticket now. I didn’t realize until I’d written this support request out that the problem likely lies with OVH, rather than Duplicati.

4

When compatibility is concerned, this is often a difficult question. I noticed after replying to you that openstack.org has an official client (OpenStackClient — OpenStack Command Line Client 6.4.1.dev15 documentation) maybe it could be interesting to give it a try.

5

How would I use the OpenStack client with Duplicati? Or are you suggesting I should use it instead of Duplicati?

6

well, I’d say that you use the client according to its documentation. I freely admit not knowing a thing about openstack so I have trouble helping here.

Generally speaking, hosters provide a way to examine data stored on their servers using tested and tried software, after all, they rely on it themselves. On the other hand, programmatic access by their customers is done by developers of their customers, that is, persons whose job is protected by using advanced and complicated features always evolving and usually full of cryptic capabilities and bugs. So it is done using a different software layer constantly evolving and that often will be broken in corner cases.

When a customer is in trouble with this nice state of things, the hosters are usually never trying to find the root cause of the problem (they are all in competition and margins are not fat), they are testing with a canonical tool provided by the same guys producing their server software and if it works with this tool, it’s a problem for the customer. If it does not work with the canonical tool, it’s a problem for the guys producing their - usually open source - server software.

All this to say that the official openstack client is not a backup tool, it’s a tool to certify that a server implementation is conforming. Being user friendly is a plus but not mandatory. Openstack is providing a bunch of tools, I have only given you the only one that seems supported - all others are using python 2, that is, obsolete. It is possible that there are other tools that are more welcoming but I don’t know that.

1 Like
7

Problem has been resolved after much hair-pulling. Need to specify region code, otherwise OVH gets confused. OVH must now require this parameter since last week. I would have preferred if OVH support had told me that when I asked them for help, but I’m glad this is resolved now.

1 Like
8

Thanks for sharing this. In some cases Duplicati could give better information about backend problems but I don’t see this happen for Openstack in next Duplicati.

9

I think the only thing Duplicati could do is mark the Region Code as required. Thank you for your advice/ideas; they were much more useful than OVH support.

10

It’s not required in all cases; so it would be necessary to mark it as required only for some providers. I am not sure that there is an easy way to do that (easy as in not having to hack at the code, just setting config variables)

11

Maybe provide a warning next to the button and have the user decide?

Note: Some providers may require this parameter.