The backup storage destination is missing data files

Support
1

Failed: The backup storage destination is missing data files. You can either enable --rebuild-missing-dblock-files or run the purge command to remove these files.

I’ve added the suggest option. Then the backup:

Failed: The database was attempted repaired, but the repair did not complete. This database may be incomplete and the backup process cannot continue. You may delete the local database and attempt to repair it again.

I’ve deleted and re-generated the local database:

Failed: Found 3 files at the remote storage, but none that could be parsed.

I’ve run purge command.

ErrorID: PurgeNotAllowedOnIncompleteDatabase

The purge command does not work on an incomplete database, try the purge-broken-files operation.

Return code: 100

I’ve run the purge-broken-files command.

ErrorID: CannotListOnDatabaseInRepair

Cannot continue because the database is marked as being under repair, but does not have broken files.

Return code: 100

And now?

1 Like
2

Ouch, that looks painful.

What version of Duplicati are you using, on what OS, and to what destination?

Did you happen to verify on the destination if the reportee missing files really weren’t there?

3

Duplicati - 2.0.3.13_canary_2018-10-31
Windows 8.1
destination folder: net folder assigned to drive L: - see note (*)
How can I list the missing files? I don’t remember how to do…

(*) The error appears when I move the pendrive used as the backup destination from the usb port of my notebook to the usb port of my router; anyway, I assign to the net resource the same drive letter (L:) assigned to the previous usb location. I tried a restore and it seems that it is possible… but I’m unable to backup yet.

4

There is a similar case from last year where @kenkendk provided some advice but we never heard back from the user

There’s also this thread

What types of files are on the remote storage? It is a complete backup?

5

Unfortunately the suggested operation (delete or rename the dlist file) didn’t work.
I’m unable to understand your last question. Does “remote storage” refer to source or destination path? The files in the source are mainly text and doc files and jpg files. The files on the destination are named duplicati-xxx.dblock.zip.aes and duplicati-xxx.dlist.zip.aes.

6

Remote storage is destination. Okay, so the destination seems to hold a “full” backup, but the fact that they can’t be parsed is odd.

7

It sounds like initially one ore more dblock file was detected missing (hence the rebuild-missing-dblock-files or purge suggestion in the error message).

@rinaldi6109, if you look at the destination do you see more than 3 files? If so, then my guess is your job is not pointing to the correct destination folder (since the initial error message mentioned finding only 3 files).

8

I didn’t realize it. No, I have a lot of .dindex. and .dblock. files.

I’ve deleted and re-generated the local database another time and I got:

Failed: Recreated database has missing blocks and 1 broken filelists. 
Consider using "list-broken-files" and "purge-broken-files" 
to purge broken data from the remote store and the database.

After doing list-broken-files, I got:

0 : 11/11/2018 06:46:10 (16912 match(es))
C:\Users\Antonio\Desktop\Brave.lnk (2,14 KB)
C:\Users\Antonio\Desktop\Firefox FF4.RAMdisk.lnk (1,28 KB)
C:\Users\Antonio\Desktop\Firefox FF4.lnk (1,99 KB)
C:\Users\Antonio\Desktop\Firefox FF5.lnk (1,99 KB)
C:\Users\Antonio\Desktop\Firefox def1.RAMdisk.lnk (1,97 KB)
... and 16907 more, (use --full-result to list all)
Failed to send message: System.NotImplementedException: Operazione o metodo non implementato. => Operazione o metodo non implementato.
Return code: 0

I’ve counted manually the files in the source folders and they are about 16900. I fear that all of them appear as broken…

9

Probably not. The files on the destination are compressed files and don’t directly correctly to the broken files (such as brave.lnk in your example). It’s quite possible all 16,912 broken files fit in just one or two of the destination dblock.zip.aes files. :slight_smile:

What the list-broken-files message is telling you is that (at least those first 5) files associated with your 11/11/2018 backup are broken. To see if other versions (backup dates) are also affected you’d have to use the --full-result parameter and see if any other lines like that first "0 : 11/11/2018 06:46:10 (16912 match(es)) " one exist.

You might also want ot look at How to test / verify your backups if you want to double check things.

10

Listing broken files with full-result option gives:

0 : 11/11/2018 06:46:10 (16912 match(es))
C:\Users\Antonio\Desktop\Brave.lnk (2,14 KB)
C:\Users\Antonio\Desktop\Firefox FF4.RAMdisk.lnk (1,28 KB)
C:\Users\Antonio\Desktop\Firefox FF4.lnk (1,99 KB)
C:\Users\Antonio\Desktop\Firefox FF5.lnk (1,99 KB)
C:\Users\Antonio\Desktop\Firefox def1.RAMdisk.lnk (1,97 KB)
... and 16907 more, (use --full-result to list all)
Failed to send message: System.NotImplementedException: Operazione o metodo non implementato. => Operazione o metodo non implementato.
Return code: 0
11

Yet Duplicati only sees 3 files, and doesn’t like the filename design. I too wonder if it’s not pointed to backup destination. Are you running Duplicati as a Tray Icon (not Windows service) in the same user session where drive letter L: shows many files? Please clarify. You can use your job’s Commandline option to see if “Target URL” looks like drive letter L:. Although it should say the same thing as Windows Explorer or “dir” command, you can also take the Target URL you just saw, and place it in double quotes on a Command Prompt (then if there is a backslash before the last quote, delete or double it). Here’s an example using a small test backup:

C:\>"C:\Program Files\Duplicati 2\Duplicati.CommandLine.BackendTool.exe" list "file://C:\Duplicati Backups\local test 1"
Name    Dir/File        LastChange      Size
duplicati-20181020T002117Z.dlist.zip    File    10/19/2018 8:21:18 PM   1.43 KB
duplicati-20181113T001826Z.dlist.zip    File    11/12/2018 7:18:29 PM   1.69 KB
duplicati-20181113T005003Z.dlist.zip    File    11/12/2018 7:50:05 PM   1.69 KB
duplicati-b4fd07fcfb03a4bebb04a2774d261b148.dblock.zip  File    11/12/2018 7:18:28 PM   15.19 MB
duplicati-bcabc416ce9194a87a7dcb95703f542dd.dblock.zip  File    10/19/2018 8:21:17 PM   5.13 KB
duplicati-bf1f56edd72144613ba96bb22cc1546c7.dblock.zip  File    11/12/2018 7:50:04 PM   511 bytes
duplicati-i1b3b0a8387f94e479c2cc49acc1204be.dindex.zip  File    10/19/2018 8:21:18 PM   1.23 KB
duplicati-i407d46fdb51744a4945b5d082f3aa635.dindex.zip  File    11/12/2018 7:18:28 PM   11.64 KB
duplicati-iba52b91b83d84f9f84598e71ae717ff6.dindex.zip  File    11/12/2018 7:50:04 PM   567 bytes
duplicati-verification.json     File    9/12/2018 5:55:22 PM    586 bytes

C:\>

Before growing too concerned about what’s broken in the backup, make sure you’re pointing at the backup.

12

If I made something wrong, I’m unable to say what:

E:\>"C:\Program Files\Duplicati 2\Duplicati.CommandLine.BackendTool.exe" list "file://L:\Backup\duplicati\documenti\"
Command failed: Unwrapped TargetInvocationException
System.Exception: Unwrapped TargetInvocationException ---> System.ArgumentExcept
ion: Caratteri non validi nel percorso.
   in System.IO.Path.CheckInvalidPathChars(String path, Boolean checkAdditional)

   in System.IO.Path.IsPathRooted(String path)
   in Duplicati.Library.Backend.File..ctor(String url, Dictionary`2 options)
   --- Fine della traccia dello stack dell'eccezione interna ---
   in Duplicati.Library.DynamicLoader.BackendLoader.BackendLoaderSub.GetBackend(
String url, Dictionary`2 options)
   in Duplicati.CommandLine.BackendTool.Program.RealMain(String[] _args)
13

Please delete the backslash before the last quote. I think doubling also works, but a single backslash doesn’t.

EDIT: Also need clarification on how the regular Duplicati is run. Is it started automatically at Windows start, or automatically at login of this user. Is Windows Command Prompt used for it (prior to the current test request)?

14

Ok. This is the output:

|Name|Dir/File|LastChange|Size|
|---|---|---|---|
|duplicati-b797ba22bb5a0435dbc9ada747e346b84.dblock.zip.aes|File|06/08/2017 12:02:32|49,99 MB|
|duplicati-i7624934963c945cba310300a4e992c69.dindex.zip.aes|File|06/08/2017 12:02:36|32,53 KB|
|duplicati-b4f6482b8e4134074ad22eaf0dbd3eb4a.dblock.zip.aes|File|06/08/2017 12:02:42|20,02 MB|
|duplicati-i9dd3bdf77a4e49a78a171e610680985e.dindex.zip.aes|File|06/08/2017 12:02:46|20,61 KB|
|duplicati-bbd2d90f04de844b1b189ffb3fc0e30bb.dblock.zip.aes|File|07/11/2018 17:07:24|218,87 KB|
|duplicati-ib4e591d2491140468241dd728bc4e503.dindex.zip.aes|File|07/11/2018 17:07:24|1,65 KB|
|duplicati-b3ff9da1c39924c959c1afc98616ed3a7.dblock.zip.aes|File|25/12/2016 11:32:28|49,90 MB|
|duplicati-ic6a1a51366744d878889eeea32be4eff.dindex.zip.aes|File|25/12/2016 11:32:28|57,34 KB|
|duplicati-bea126136e8574d95b5bca003be1700f3.dblock.zip.aes|File|11/11/2018 07:47:02|136,33 KB|
|duplicati-b1c8c644e1b9c4beba601cb0e1f0a0d45.dblock.zip.aes|File|25/12/2016 11:32:50|49,94 MB|
|duplicati-if7a08067152b41b882b5ed5af23726bd.dindex.zip.aes|File|25/12/2016 11:32:50|47,36 KB|
|duplicati-ba64d9543be3d417c87ecdf14dadd7a18.dblock.zip.aes|File|25/12/2016 11:32:54|50,00 MB|
|duplicati-i8f5cb16df7924b488528aa7305e334a8.dindex.zip.aes|File|25/12/2016 11:32:54|30,54 KB|
|duplicati-b50f4b72237804c698d8d269dfb17dda8.dblock.zip.aes|File|25/12/2016 11:33:00|49,93 MB|
|duplicati-ief58d9295bc54e8aae94d90f2c7e2acd.dindex.zip.aes|File|25/12/2016 11:33:00|43,98 KB|
|duplicati-b0905ac2a865f44f197843ab30b4268df.dblock.zip.aes|File|25/12/2016 11:33:04|49,94 MB|
|duplicati-i775ba77e63d043519ac3468fe0d6ef8a.dindex.zip.aes|File|25/12/2016 11:33:04|21,59 KB|
|duplicati-b6f3fb57576af42d2808672dd614673ba.dblock.zip.aes|File|25/12/2016 11:33:14|49,99 MB|
|duplicati-i5da3287ffbce41daa5eb95cd6b2bc508.dindex.zip.aes|File|25/12/2016 11:33:14|17,97 KB|
|duplicati-b84490b2fa7a144f7baad4896034737b6.dblock.zip.aes|File|25/12/2016 11:33:18|49,99 MB|
|duplicati-iade917c7e12e4cbfabd13cba47f49d89.dindex.zip.aes|File|25/12/2016 11:33:18|17,97 KB|
|duplicati-b9694e9310bc14f89ae152ff7aaa05d9f.dblock.zip.aes|File|25/12/2016 11:33:22|49,95 MB|
|duplicati-if89772321a15426189dffcab09f58391.dindex.zip.aes|File|25/12/2016 11:33:24|80,62 KB|
|duplicati-b6cedeeb7817e4026a1701952a7564745.dblock.zip.aes|File|25/12/2016 11:33:34|49,96 MB|
|duplicati-i116c07149bd0419c9518d50dc582dd2e.dindex.zip.aes|File|25/12/2016 11:33:34|67,51 KB|
|duplicati-be3763f53c5cd49d4957c0a8fbfa083ee.dblock.zip.aes|File|25/12/2016 11:33:44|49,97 MB|
|duplicati-i774ecf3783f94ebea6ac4b41a9e91977.dindex.zip.aes|File|25/12/2016 11:33:44|50,87 KB|
|duplicati-b939fa278a38644fc97ec85e626ea70d5.dblock.zip.aes|File|25/12/2016 11:33:48|49,91 MB|
|duplicati-i3883c20df0e3487f98d333269c206943.dindex.zip.aes|File|25/12/2016 11:33:48|46,43 KB|
|duplicati-b7da2c83010e1462b9e9288a75b5fd4bf.dblock.zip.aes|File|25/12/2016 11:34:00|50,00 MB|
|duplicati-i6eb7f8cee51e4647965805715be4337a.dindex.zip.aes|File|25/12/2016 11:34:00|57,36 KB|
|duplicati-20181111T054610Z.dlist.zip.aes|File|11/11/2018 07:47:02|1,67 MB|
|duplicati-b4bf30c06a0af4885a26bc4c128db40d5.dblock.zip.aes|File|14/08/2017 04:05:56|49,98 MB|
|duplicati-bc7d1a464cf5a4297a2a199afd1aacfdc.dblock.zip.aes|File|25/12/2016 11:34:16|49,92 MB|
|duplicati-ibee4737f5e2e4e809d216febd7a7cf5b.dindex.zip.aes|File|25/12/2016 11:34:16|37,98 KB|
|duplicati-b5ef9b4b669174ff982316f8533619eaf.dblock.zip.aes|File|25/12/2016 11:34:22|49,99 MB|
|duplicati-i6f39c71b5412474196684596569fb8c6.dindex.zip.aes|File|25/12/2016 11:34:22|42,15 KB|
|duplicati-b71cc72d54557472eac7d8a9184af5b5d.dblock.zip.aes|File|25/12/2016 11:34:28|49,91 MB|
|duplicati-i858c9da315e749c68cba6ec3e4c9550c.dindex.zip.aes|File|25/12/2016 11:34:28|39,53 KB|
|duplicati-bf7db98f66a144e3da3d9dbcef7c8c044.dblock.zip.aes|File|25/12/2016 11:34:34|50,00 MB|
|duplicati-i1f57571569df45d4a669b60f5c9dac68.dindex.zip.aes|File|25/12/2016 11:34:34|39,31 KB|
|duplicati-bfc35c9fd351748e5888dd06b981de048.dblock.zip.aes|File|25/12/2016 11:34:40|49,95 MB|
|duplicati-i622481aab6e3487daf5c66275dd60cf7.dindex.zip.aes|File|25/12/2016 11:34:40|40,56 KB|
|duplicati-b1b2bb099652f4f0383b1d08d0dc955ff.dblock.zip.aes|File|25/12/2016 11:34:46|49,91 MB|
|duplicati-i54630187441949bba1c4326ba328669a.dindex.zip.aes|File|25/12/2016 11:34:46|38,84 KB|
|duplicati-b56572f5af71e4bb8981d890c8767efad.dblock.zip.aes|File|25/12/2016 11:34:50|49,99 MB|
|duplicati-ia3b6a923c3874e2d8c28b51ebb5ec596.dindex.zip.aes|File|25/12/2016 11:34:52|38,23 KB|
|duplicati-b679c583e3a224ac4a7e642c1e49f4f6c.dblock.zip.aes|File|25/12/2016 11:34:56|49,96 MB|
|duplicati-i4463efcdcc404022b2f56a8e491cf8e2.dindex.zip.aes|File|25/12/2016 11:34:56|36,36 KB|
|duplicati-bd334ebc0711d4c828508c4b7b54e1d4c.dblock.zip.aes|File|25/12/2016 11:35:02|49,93 MB|
|duplicati-ic496e675e0974f87bfb0ee64ddca3efb.dindex.zip.aes|File|25/12/2016 11:35:02|36,17 KB|
|duplicati-b80aaf5f96ba748209e8febbecaac8c3d.dblock.zip.aes|File|25/12/2016 11:35:06|49,96 MB|
|duplicati-ib9345e67bbd041f3a70ad35e8c2e6493.dindex.zip.aes|File|25/12/2016 11:35:08|38,65 KB|
|duplicati-b3c2a0c8453df4113a006a46d539422d3.dblock.zip.aes|File|25/12/2016 11:35:12|49,92 MB|
|duplicati-i4d56d1001d5240ea9d55cfa39f193459.dindex.zip.aes|File|25/12/2016 11:35:12|38,15 KB|
|duplicati-bea76d31fb802459d9d12e642ca173e9f.dblock.zip.aes|File|25/12/2016 11:35:18|49,93 MB|
|duplicati-i5c7d4e14f31c447d9c53c1fc6acead3e.dindex.zip.aes|File|25/12/2016 11:35:18|47,04 KB|
|duplicati-bcb86d84f7ee44654baa1ef655115e2ab.dblock.zip.aes|File|25/12/2016 11:35:24|49,97 MB|
|duplicati-ifba078b731fc4df08bc51b9bad8bc37c.dindex.zip.aes|File|25/12/2016 11:35:24|58,33 KB|
|duplicati-b5332ce2647414f2aa4a03faea2d9632d.dblock.zip.aes|File|25/12/2016 11:35:32|49,97 MB|
|duplicati-i6e1df2226a6d45a78bfcf92620b18ef0.dindex.zip.aes|File|25/12/2016 11:35:32|54,14 KB|
|duplicati-bdf27576706b24f1193b1d67756be55e2.dblock.zip.aes|File|25/12/2016 11:35:38|49,95 MB|
|duplicati-i1d7a1df5ec5e4eb68f6c9d655a26e99b.dindex.zip.aes|File|25/12/2016 11:35:38|51,33 KB|
|duplicati-b90def0b6fe93486a8e6bc9c4fa6de1b2.dblock.zip.aes|File|25/12/2016 11:35:46|49,98 MB|
|duplicati-i208695e8d21f4ac5ad9529914c6665da.dindex.zip.aes|File|25/12/2016 11:35:46|54,37 KB|
|duplicati-bd499e073532c41648b748487d2c419d4.dblock.zip.aes|File|25/12/2016 11:35:50|49,93 MB|
|duplicati-i5f7a33bc1d1a4637b8199cb3699aef4c.dindex.zip.aes|File|25/12/2016 11:35:50|53,89 KB|
|duplicati-b72c91e1d046349429f9f2bd25b6e4975.dblock.zip.aes|File|25/12/2016 11:35:56|49,92 MB|
|duplicati-i2eeeb01755b74db588ee618193634e51.dindex.zip.aes|File|25/12/2016 11:35:56|52,95 KB|
|duplicati-b55cf140c4f794776aa2cb6a796ce8c95.dblock.zip.aes|File|25/12/2016 11:36:02|49,95 MB|
|duplicati-i63a9a946086048578116eddc0ad8e34f.dindex.zip.aes|File|25/12/2016 11:36:02|50,06 KB|
|duplicati-bff52f5939ea24288b8cf9441c6313529.dblock.zip.aes|File|25/12/2016 11:36:08|49,94 MB|
|duplicati-ifdcc68357f5c404d8abe929f47d15ddc.dindex.zip.aes|File|25/12/2016 11:36:08|52,03 KB|
|duplicati-b888f0751c61e48448ac65d791a16fae7.dblock.zip.aes|File|25/12/2016 11:36:16|49,97 MB|
|duplicati-i5aad5e37ebc344509bd9f192b920cac9.dindex.zip.aes|File|25/12/2016 11:36:16|50,67 KB|
|duplicati-b702427fbb4b54074b05a814fd831cb18.dblock.zip.aes|File|25/12/2016 11:36:24|49,92 MB|
15

So we now know that a Duplicati from a Command Prompt can see the files. Not a big surprise, but I still need to know how the regular Duplicati is run. If a Windows service, it doesn’t have the same view. Best to hear would be to know that you used the default install, have a Duplicati Tray Icon in the lower right corner of your screen, and can click on that, “Quit”, then restart Duplicati from a desktop shortcut or something. Might as well do so anyway.

EDIT: and a good next step after that (assuming the path works and this isn’t a Windows service situation) is run MENU --> Restore --> Direct restore from backup files --> Next --> point to it, and see if you can see source files. Doing it this way bypasses any problems the local database may have, and if it works you can relax somewhat…

16

Yes to all:

  1. I have Duplicati tray icon;
  2. I can quit clicking on it and restart Duplicati from an entry in the “menu” (classic shell)
  3. I have successfully completed a direct restore.

P.S.: at the end of the (successful) direct restore, I got:
Recreated database has missing blocks and 1 broken filelists. Consider using "list-broken-files" and "purge-broken-files" to purge broken data from the remote store and the database.

17

I’m not sure if the restart helped, but it might have, otherwise I wonder if Duplicati had an old L: or something. Did you happen to save copies of any old databases for this job (database path is visible on job’s Database option)? The database made for direct restore is faster to build because it’s partial. I don’t know what to say about the error, except I think the database is also temporary, and if so then there’s not much to look at now.

I’m not sure everything is healthy, but I’m glad a direct restore seemed to work, in spite of getting a message. The question now is how best to recreate the database (unless you saved a copy). For performance, moving the pendrive back might be faster (I’m not certain how much), and a Duplicati restart wise unless we can find out what went wrong with the view of L: originally. The coarse, possibly slow way of doing database recreate is probably much like what you did earlier, except we hope it sees more files and runs for maybe quite awhile.

A more selective manually-run method is to use The REPAIR command repeatedly with –dbpath pointing to a nonexistent file each time, and varying the –version option to see which versions appear error-free. The most recent version is 0 (the Restore UI shows the version number to the left of the date). You can use commas or dashes to do version lists and ranges if you like. The goal is to recreate your database to the extent possible.

18

No, I haven’t any database backup. The method you suggest is too costly for me.
Anyway, thank you for your support.
I’m thinking to keep a copy of the backup archives and configure a new one in a new folder, since I have realized that I cannot simply delete the backup files in the destination folder and launch the old job.

19

I’ve solved my issue.
It happened that I tried to run the backup job with another usb pendrive (after having copied all the backup archives on it), and the backup job completed with no problem.
So I argued that the issue did depend on some characteristic of the pendrive, and indeed I recognized that the first one was formatted as exFat instead as Fat32 like the second one, while my router does not support such file system. For some strange reason, I did can browse and copy its content, but evidently Duplicati does some operation that my router cannot perform regularly on exFat partition.
All’s well that ends well!

20

Good catch on figuring that out!

If you feel like testing… the next time your exFAT drive is connected to the router, try creating a new file - I’m wondering if your router supports read-only exFAT access.