Found 33 files that are missing from the remote storage, please run repair

Support
1

Hi,

I have a backup set going back to 2021 and it’s now started with the dreaded error:

“Found 33 files that are missing from the remote storage, please run repair”

I have tried the following without being able to fix it.
Running the backup again.
Deleting recent backup sets.
Repairing the database.
Deleting and recreating the database.
Running “list-broken-files” - which comes back with nothing.

It’s backing up to the same local FTP as it’s always done.

None of the above has worked. Is there something else I can try to help recover this dataset?

As the files are missing from remote storage, is there anyway to force a full backup etc?

Thanks

2

That was said confidently. Does that mean you got names (if so, are they dindex or dblock), and looked? Duplicati has the unfortunate habit of throwing away logs on failure, which is when you’d want error logs. There might be improvements coming. Meanwhile, look at About → Show log → Live → Warning in run.

There’s also a fix coming for a bug that causes Missing file claims due to bad records. Yours is different, however it would be worth thinking about the history preceding this. Check the logs for gaps and errors. Server log at About → Show log → Stored might also give some clues as to what came before the error.

What OS is this?

So I guess it’s accessible to look at if needed. Is it attached to an OS where Python scripts might be run?

3

Thanks for the reply.

I was only going on the info from Duplicati!

Windows 11 is where the backups run from

Yeah, I can run Python on the server.

4

Thanks for the initial reply. The rest is still needed, as information on missing files is severely lacking.

If you like, a database bug report can be made, uploaded someplace, and linked. Possibly it will help.

We can save it for last, after the other steps, but I have some unofficial backup integrity checker tools, basically looking for things that might block a database recreate. I’d prefer looking at the damage first, however if you get an urge to try recreate, please save current database for history, or if recreate fails.

Or it might be too late, as

might have lost old database and the job logs inside it. What errors came from the repair and recreate?

Are the FTP files encrypted? The checker would work off a decrypted copy of the dlist and dindex files.

5

I should be able to recover the database from another backup if required.

The repair and recreate seems to run without errors. But once it’s done, the next backup fails with the original error of missing files.

The files on the FTP server are encrypted using Duplicati.

6

Be really careful with that. A stale database will see new destination files as extra. Repair will delete.

Similarly, missing files errors come from a comparison of the database records to what’s now visible, however deleting the database deletes all records so if recreate worked, it should match destination. Maybe there’s a similar issue with list-broken-files not giving error if run on recreated database.

Surprising. Are there old logs in there as well? But back to the long-past question – what sort of files?
Repair from an intact database should fix missing dindex by uploads, but database state is unknown. Similar treatment for dlist files. The hard ones are dblock files, which are actual data from the source.

7

The files that are being backed up are mainly photo files.

I can’t help but feel new files were added to the backup source as a backup was being run which may have caused the original error, but i can’t be sure.

8

If other backup DB is newer than the newest Duplicati file on the destination, it might be a useful tool.
If nothing else, perhaps it will have job logs (best is if it’s scheduled, so misses are clear), and errors.

That should be no problem, except that some of them might not have made it into a backup just then.

This is not making progress, as questions aren’t being answered. 33 missing files is not enough data.

9

I’ll have to check but I’m not 100% sure that deleting the database in the web gui is actually deleting it. So that might be why i get the same error?

10

I think it asks using a confirmation popup, then deletes. If you see old logs, it didn’t.
There are some cases where it keeps a backup copy in case of failure, but I’m not
sure this is one of those cases. It might be during DB version upgrade. Not certain.

Regardless, check for job logs. If you know expected times, see if any are missing.
Server log at About → Show log → Stored keeps clues on attempts that fully failed.
Live log at About → Show log → Live → Warning will reveal names of missing files.
Severity of problem depends on what those are.

11

Ok thanks.

I’ll see what I can find out from the logs but it might not be until next week until I get a chance to properly look.

I’ll update here once I have.

Thanks for your help so far.

12

This procedure seems backward. After seeing the message ‘missing files’, list-broken-files should have been run first. It may be the reason it finds nothing.
I have given a look to this function and ‘normally’ (if an enterprising user has not frankensteined a different database on the backend, for example), it is listing correctly the source files that are recorded in the database as backed up but are also recorded as saved in block files missing from the backend.

13

Order does matter. I picked on DB recreate as a way to hide problem that might exist.
Ideal order may vary and is reliant on Duplicati giving info, and user knowing to use it.
The manual is a bit light on troubleshooting and is currently wrong on list-broken-files.

Maybe, maybe not. It depends on what’s missing, but Duplicati makes that hard to see.
I’m hoping that the Improve result reporting PR will at least save the job log after error.
VerifyRemoteList() logs Missing and Extra at Warning level, so job log might get those.

list-broken-files has the same problem, ever since it moved to the current logging
system where Information level is off by default, thus suppressing useful information.
This is inherently contradictory for tools whose purpose is to provide information to you.

Recovering by purging files in the manual does not work as shown, ever since change.

It’s also not good at reporting or replacing missing dlist and dindex files. For missing, it
might give you a count (if you set log level right), then say to run purge-broken-files.

That run then logs claims of what it did, but if you look at the database, it didn’t actually.

Test case: Backup a short file, hide dlist and dindex, play around with those commands.

In comparison, Repair will reupload missing dlist and dindex (assuming good database).

If you have a for-sure missing dblock, the broken-files commands are the better options.

14

here is the output of list-broken-files after removing a dblock file
I have marked the lines that are hidden by not having the Information level, the lines without the stars are the default output. What are the ones that in your opinion are missing and are vital to understanding what is happening ? It would be easy and riskless to change the Information to Warning in the code (that is it could be in next Canary without problem)

**The operation ListBrokenFiles has started**
**No broken filesets found in database, checking for missing remote files**
**Backend event: List - Started:  ()**
**  Listing remote folder  ...**
**Backend event: List - Completed:  (105 bytes)**
**Marked 1 remote files for deletion**
4       : 02/08/2023 17:11:59   (7 match(es))
        C:\Users\gp\duplicati\Duplicati\GUI\Duplicati.GUI.TrayIcon\bin\Debug\Duplicati-server.sqlite (72,00 KB)
        C:\Users\gp\duplicati\Duplicati\GUI\Duplicati.GUI.TrayIcon\bin\Debug\Duplicati.debug.log (8,69 MB)
        C:\Users\gp\duplicati\Duplicati\GUI\Duplicati.GUI.TrayIcon\bin\Debug\QSVDHHPCMR.sqlite (14,78 MB)
        C:\Users\gp\duplicati\Duplicati\GUI\Duplicati.GUI.TrayIcon\bin\Debug\QSVDHHPCMR.sqlite-journal (2,79 MB)
**Return code: 0**

My point of view is that the main problems in the default output are that:

  • the ‘4’ is the fileset number (it’s easy to understand, if you know Duplicati)
  • the 7 vs 4 discrepancy is because the 7 refers to the total number of file entries, while the 4 refers to
    the files, that is, there are 3 directories that are backed up in the missing block.

But I don’t see the really vital information that we are missing in the extra lines.

Edit:
I forgot to say that displaying

Marked 1 remote files for deletion

in the default output in list-broken-files would be false information. List-broken-files does not mark remote files for deletion. Purge-broken-files does, but both functions use common code.

15

I don’t know in this case, however I take the opportunity to say that this change, while a clear progress by itself, has a nasty side effect of turning information function like list-broken-files whose purpose is information about problems into ‘failing’ functions. Before the next release I intend to revisit this code to turn the output into ‘error’, not ‘failing’ when something wrong is found. The function has succeeded in finding problems, it has not failed.

16

I might have to walk that part back – I’m looking further into the source code now.

We did lose a lot of context though, and making it Warning may cause unwanted
side effects similar to your follow-up post. I’m not sure how best to solve log level.

The other question is whether this is expected to name files causing brokenness.
Just looking at the manual, I see a message that probably is related to this code:

and if so, it’s at Warning level and probably deserves that, and probably is shown.

In interrupt test, occasional Missing and Extra are probably by VerifyRemoteList()
which logs (at Warning level) the results of RemoteListAnalysis() which isn’t vocal.

ListBrokenFilesHandler.cs goes directly to RemoteListAnalysis, so says little.
Arguably if its goal is to give the end result rather than its cause, it might be OK…

Although we’d rather not have to use them much, good repair data and tools help.
Duplicati can be a nuisance to fix when it breaks, as exemplified by the topic here.
Over time, perhaps things can be improved, so that UX and support load is better.

17

you mean the remote files right ? If yes, given that these files are really non existing, I’m not sure what to do about them. Sure it’s a problem that they don’t exist, but lacking any proof that it’s the result of a bug and not a network or backend failure, there is not much to go on. If it’s for debugging purpose, the missing remote file name is by default in the full log when the backup fails.

18

Yes, the problem behind the problem.

If you start with Repair and an intact database, it puts them back, even with original file names.

I’m not sure what full log that is. I know you can get it with work as requested, but not by default.
Maybe we’ll learn something tomorrow. I don’t know if system is inaccessible weekends or what.

19

I did not try that.
However, the OP did:

Running the backup again.
Deleting recent backup sets.
Repairing the database.
Deleting and recreating the database.
Running “list-broken-files” - which comes back with nothing.

the first step, running again the backup, could hardly have been anything else than a no-op.
Why ‘deleting recent backup sets’ would have changed the outcome of ‘repairing the database’ I don’t know.
What I was mostly objecting was ‘Deleting and recreating the database.’ before running ‘list-broken-files’ though.

20

So this is strange.

I’ve made a copy of the backup files on the server and recreated the profile so I’m not messing with the original files.

I run a repair and this is successful. I’d I then run a backup, it fails. The logs say that certain files are missing on the server and to run a repair. Which once done ends up with the same missing files result.

If I change the destination from FTP to SFTP (SSH) and repair and run the backup, it succeeds. If I then change the destination back to FTP, it fails with the missing files error again.

I’ve checked a file the logs says is missing from the server. It does exists and the permissions seem correct. But when FTP is used, it seems to think those files aren’t there.