How to list / purge broken files

How-To
1

This is a first crack at helping people understand what will happen with list-broken-files and purge-broken-files commands. It is by no means complete, but it IS a wiki, so feel free to add / update anything you feel might be useful to other users. :slight_smile:

While Duplicati is very robust in handling errors (including missing destination files) there may come a time when you need to remove references to backup versions from your job.

When this happens, the use of the list-broken-files command will list what what versions of what files will be REMOVED from your backups IF you run the purge-broken-files command.

As of the 2.0.3.7 canary (and likely very much older) version, list-broken-files can be run with:

  1. Select the job “Commandline …” menu

  2. Select a “Command” of list-broken-files

  3. Replace all “Commandline arguments” with --full-result

    If this is too long, consider instead using --console-log-level=verbose to get a short list of SOME of what will be affected. Alternatively, keep using --full-result but also use --log-file=<path> to have the full output send to the log file. (Be sure to put each commandline argument on it’s own line.)

  4. Try running the command with all “Advanced options” in place. If it doesn’t work, remove items like the exclude and include filters, but be sure to LEAVE the dbpath and either no-encryption or passphrase (and possibly encryption-module) parameters

  5. Click the ‘Run “list-broken-files” command now’ button at the bottom of the page

Note: Because this is LIST-broken-files, not PURGE-broken-files, even though some message below mention marking remote files for deletion that does NOT actually happen.

Once you’re satisfied with the list-broken-files results, just do the same steps as above but with selecting purge-broken-files in step 2.

Below are samples of list-broken-files when used with --log-level=verbose on a SMALL test backup

No missing files (working backup) 
The operation ListBrokenFiles has started
No broken filesets found in database, checking for missing remote files
Backend event: List - Started:  ()
  Listing remote folder ...
Backend event: List - Completed:  (24 bytes)
Skipping operation because no files were found to be missing, and no filesets were recorded as broken.
Return code: 0
Purposefully removed `dblock` file (broken backup) 
The operation ListBrokenFiles has started
No broken filesets found in database, checking for missing remote files
Backend event: List - Started:  ()
  Listing remote folder ...
Backend event: List - Completed:  (23 bytes)
Marked 1 remote files for deletion
1	: 6/19/2018 11:48:31 AM	(1638 match(es))
	C:\Users\Me\AppData\Local\Mozilla\updates\308046B0AF4A39CB\active-update.xml (1.15 KB)
	C:\Users\Me\AppData\Local\Mozilla\updates\308046B0AF4A39CB\updates.xml (11.57 KB)
	C:\Users\Me\AppData\Local\Mozilla\updates\308046B0AF4A39CB\updates\last-update.log (6.67 KB)
	C:\Users\Me\AppData\Local\Mozilla\updates\308046B0AF4A39CB\updates\0\update.mar (5.10 MB)
	C:\Users\Me\AppData\Local\Mozilla\updates\308046B0AF4A39CB\updates\0\update.status (16 bytes)
	 ... and 1633 more, (use --full-result to list all)
0	: 6/19/2018 11:55:18 AM	(1638 match(es))
	C:\Users\Me\AppData\Local\Mozilla\updates\308046B0AF4A39CB\active-update.xml (1.15 KB)
	C:\Users\Me\AppData\Local\Mozilla\updates\308046B0AF4A39CB\updates.xml (11.57 KB)
	C:\Users\Me\AppData\Local\Mozilla\updates\308046B0AF4A39CB\updates\last-update.log (6.67 KB)
	C:\Users\Me\AppData\Local\Mozilla\updates\308046B0AF4A39CB\updates\0\update.mar (5.10 MB)
	C:\Users\Me\AppData\Local\Mozilla\updates\308046B0AF4A39CB\updates\0\update.status (16 bytes)
	 ... and 1633 more, (use --full-result to list all)
Return code: 0
Purposefully removed `dlist` file 
The operation ListBrokenFiles has started
No broken filesets found in database, checking for missing remote files
Backend event: List - Started:  ()
  Listing remote folder ...
Backend event: List - Completed:  (23 bytes)
Marked 1 remote files for deletion
No broken filesets found
Return code: 0
Purposefully removed `dindex` file 
The operation ListBrokenFiles has started
No broken filesets found in database, checking for missing remote files
Backend event: List - Started:  ()
  Listing remote folder ...
Backend event: List - Completed:  (23 bytes)
Marked 1 remote files for deletion
No broken filesets found
Return code: 0
Purposefully zero sized `dblock`, `dindex`, or `dlist` file 
The operation ListBrokenFiles has started
No broken filesets found in database, checking for missing remote files
Backend event: List - Started:  ()
  Listing remote folder ...
Backend event: List - Completed:  (27 bytes)
remote file duplicati-b4a7ed4aae4e8452e95db1738320ff601.dblock.zip is listed as Verified with size 0 but should be 52357878, please verify the sha256 hash "ZVJ2lnQdRJxxoCE81IkYS2M4lVWjMcVxzfTwtKhxgPg="
Skipping operation because no files were found to be missing, and no filesets were recorded as broken.
Return code: 0
2

IMHO, if this is required to get it to work, then such replacement should be handled automatically when picking the “list broken files” option from the dropdown. As a stupid user, I would never have found my way to this without your handholding (which is appreciated).

(Also, i still get the same result I just mentioned in the other thread, even when doing it this way).

Edited to add:
I think the following is probably incorrect in a majority of cases:

Leave the “Advanced options” alone (or delete them, it shouldn’t make a difference)

If you have any filtering / custom exclusions (so… >99% of setups?), the command will fail citing “filtering not allowed”.

3

Thanks for the feedback - I’ve updated the post to default to suggesting removal of all advanced options.

I don’t disagree, and there are actually a few other command line commands that could benefit from this sort of pre-cleaning.

Unfortunately, we just haven’t found any development time to spend cleaning up the rough edges like this. :frowning:

I haven’t actually looked at this bit of code yet myself, but maybe it would make a good Giithub ticket with a “good first issue” tag for somebody wanting to get started helping out.

Oh, and I’m glad the instructions (eventually) worked for you - thanks for helping to make them better for the next person!

4

Not to be a dark cloud, but to clarify on my last message, i haven’t actually gotten this working yet. Every permutation i’ve tried (including the ones you’ve suggested) result in the same failure so far.

Edit: fwiw, it fails to work for both my Google Drive backup job as well as my B2 backup job. And in both cases I leave the “target url” box untouched, all my permutations have been in the command line / advanced option boxes.

5

Not even gonna give me a small win, eh? :slight_smile:

OK - granted, my update to “remove all advanced options” will break it even more than leaving them all in. I’ve updated it again and tested on jobs both with and without passwords.

Basically, list-broken-files needs at a minimum:

  • list-broken-files
  • destination path
  • --dbpath
  • --passphrase OR --no-encryption (depending on how your backup is configured)
1 Like
6

I got it to work finally after your last round of advice. I did basically what i had already been trying, except when clearing out the Advanced Options i made sure to clear ONLY the “exclude” commands, and left everything else (needed or not).

7

Great! I’ll count that as a confirmation that the how-to steps are (mostly) correct now. :slight_smile:

And, yes - this is a perfect example of why it would be so much better to have the command line commands be able to ignore (or pre-filter / warn) when there are options or parameters that won’t work with the selected command. :wink:

1 Like