Stuck on repairing


#1

I’ve got an issue with Duplicati and it’s quite similair to the issues described here: Repair freezes (or takes days), but the hints are not working for me.

So in short:

  • Duplicati is missing files (more than 6000) and asks to repair.

  • Repairing takes very long. It keeps mentioning: ‘starting’ and there is some (minor) improvement in the status bar. For the first couple of hours, I also see activity in the log.

  • After a while there is no activity in the logs (for at least 3 hours), but duplicati takes a very large amount of my processor…

  • I thought it was stuck, so I cancelled it (and rebooted) and tried it again. But the same issues appear.

  • I’ve now used ‘delete and repair’, but now the following message appeared: ‘No files at remote location’.

  • Using ‘test connection’ it stated: ‘Connection Works’…

So something is not going as planned. Does some-one got some suggestions to try to figure out what’s going wrong?

BTW: I’m using the Onedrive for Buisness as my storage location.


#2

Hi @TheZim, welcome to the forum!

I’m not sure how you would have gone from 6000 files missing to “No files at remote location” unless something happened to your destination folder.

Have you tried logging into OD4B through another interface and checking if you can see the Duplicati destination folder at all?


#3

Hello @JonMikelV,
Thanks for the welcome!

Some more explanation: During an initial backup I got the message: ‘Found 6568 files that are missing from the remote storage, please run repair’. So I run Repair, and that takes fore-ever… (Or should that be with a 450Mb SQL-file?)
When I do ‘Delete and Repair’ I get the following message: ‘No files were found at the remote location, perhaps the target url is incorrect?’…
So I checked with: ‘Test Connection’, and that replies: ‘Connection Works’.

You mean with ‘another interface’ just checking if there are some files on the OD4B?. There are still files on that location.

The error file:

Duplicati.Library.Interface.UserInformationException: No files were found at the remote location, perhaps the target url is incorrect?
   bij Duplicati.Library.Main.Operation.RecreateDatabaseHandler.DoRun(LocalDatabase dbparent, Boolean updating, IFilter filter, NumberedFilterFilelistDelegate filelistfilter, BlockVolumePostProcessor blockprocessor)
   bij Duplicati.Library.Main.Operation.RecreateDatabaseHandler.Run(String path, IFilter filter, NumberedFilterFilelistDelegate filelistfilter, BlockVolumePostProcessor blockprocessor)
   bij Duplicati.Library.Main.Operation.RepairHandler.RunRepairLocal(IFilter filter)
   bij Duplicati.Library.Main.Operation.RepairHandler.Run(IFilter filter)
   bij Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action`1 method)
   bij Duplicati.Library.Main.Controller.Repair(IFilter filter)
   bij Duplicati.Server.Runner.Run(IRunnerData data, Boolean fromQueue)

So any Tips?

BTW: Before I deleted the database I made a copy. So I can reverse the steps If you would require some more information. I’m able to recreate is…


#4

Thanks for the detail! (And sorry for the delayed response.)

The “Found xxx files that are missing form the remote storage” message means that the list of files from the destination doesn’t match what Duplicati is expecting. This can be caused by things including:

  • communication error while fetching the file list (caused by things like internet connectivity issues, destination API changes that haven’t been updated in Duplicati yet, etc.)

  • actual file listing differences (caused by things like another tool or used moving / deleting destination files or renaming the destination folder / path, the destination automatically moving files to cold storage or changing ownership of the files making them no longer visible to Duplicati)

  • Duplicati’s local list of files somehow getting corrupted so it suddenly expects more files at the destination than there actually are (this is pretty unlikely as that sort of corruption will probably cause crashes rather than “polite errors”)

The Repair that you ran SHOULD have put things back in sync (basically just looking at what files where really in the destination and updating the local list to reflect that). It should not have taken “forever” - in my tests when repairing a 900MB sqlite file it took about 90 minutes.)

A database recreate (“Delete and repair”) definitely WILL take longer as it has to download more files from the destination to process into a new database AND all the database writes (basically from scratch) can take a while. (We’re already working on database optimization updates.)

This is pretty self explanatory - Duplicati isn’t finding ANY files at the destination so we’re back to:

  • communications error (should be transient and not happen every run)

  • there really are NO files there (if your “other interface” shows there ARE files there check ownership as well as the destination path)

  • Duplicati is looking in the wrong place (if the destination path got changed either in Duplicati or on the destination, even a simple case change, then Duplicati could be looking in the wrong place)

If you use the main menu Restore link and try a “Direct restore from backup files…” this should take the local file list out of the picture and let you see what’s ACTUALLY at the destination.

I’d say check that out and see what you get. If you still get nothing, then there’s probably an ownership or pathing issue.

Note that there HAVE been a few cases reported where the destination (such as Google Drive) changes ownership of a folder from belonging to Duplicati to the account user for some reason. When possible, Duplicati tries to play nice and only have access to it’s own backup folders, so if the ownership changes like that then suddenly Duplicati can’t see the files anymore.


#5

Thanks for the reply @JonMikelV,

I Will try a repair again tonight, but if it takes longer, what kind of feedback data would you require?

Using the ‘Direct restore from backup files’ it very handy! I now see that ‘testing connection’ doesn’t check the file location. If I put a bogus file destination, but the right username and password. It will state ‘Connection Worked’… This was not clear to me the first time. Is it not handy to check the location as-well?

If I select this location, I will get: “Failed to connect: No filesets found on remote target”. So probably wrong location (But I didn’t change the destination)? If I click on details I get: “Failed to connect: SQL logic error or missing database no such table: LogData”.

What is the restore function looking for? And can I check (in the log-file) which specific files on which specific location it’s looking?

Thanks in advance!

@Edit: Some extra information:
If I use the 3 dots on the 'Direct restore from backup", I will get my URL:

 od4b://nothingtosee-my.sharepoint.com/personal/nothingtosee/Documents/Backups/Laptop?auth-username=NoNamehere%40nothingtosee.com&auth-password=nothingtoseepassword

If I copy the link, without the od4b:// into my browser, I get to see all my files. But that only works when I’m logged in already… If I use a different or (incognito) browser, I will asked to log-in… I guess that should’t be right?

So I guess it something to do with the way duplicati searches for the right files?


#6

Hello @JonMikelV,

I’ve been trying to dig deeper to find the issues, but no luck. I think the communication is correct. I’ve tried it on different user, different locations, but all states that is working. But I’m not able to repair the missing files.

I’ve removed the whole backup, and tried again. I started with a partial backup (only 50% of my total files I want to backup), then there are not problems. So I’ve expanded the files to 80%, and then the same issues appears:

Found 6568 files that are missing from the remote storage, please run repair

So I was wondering what files where missing, so I tried the list-broken-files command. It took me a while to get the right command line, but I’ve ended with this commandline:

Duplicati.CommandLine.exe list-broken-files od4b://****-my.sharepoint.com:443/personal/****/Documents/Backups/Laptop****Backup --auth-username=**** --auth-password=****  --dbpath=C:\Users\****\AppData\Local\Duplicati\DMOGIOEBJY.sqlite --passphrase=**** --console-log-level=Profiling --log-file=c:/Test.txt

The responds is:

The operation ListBrokenFiles has started
Starting - Running ListBrokenFiles
Starting - ExecuteScalarInt64: INSERT INTO "Operation" ("Description", "Timestamp") VALUES (?, ?); SELECT last_insert_rowid();
ExecuteScalarInt64: INSERT INTO "Operation" ("Description", "Timestamp") VALUES (?, ?); SELECT last_insert_rowid(); took 0:00:00:00.015
Starting - ExecuteReader: SELECT "Key", "Value" FROM "Configuration"
ExecuteReader: SELECT "Key", "Value" FROM "Configuration"  took 0:00:00:00.000
Starting - ExecuteReader: SELECT "ID", "Timestamp" FROM "Fileset" ORDER BY "Timestamp" DESC
ExecuteReader: SELECT "ID", "Timestamp" FROM "Fileset" ORDER BY "Timestamp" DESC took 0:00:00:00.000
Starting - ExecuteReader: SELECT DISTINCT "B"."Timestamp", "A"."FilesetID", COUNT("A"."FileID") AS "FileCount" FROM "FilesetEntry" A, "Fileset" B WHERE "A"."FilesetID" = "B"."ID" AND "A"."FileID" IN (
SELECT DISTINCT "ID" FROM (
  SELECT "ID" AS "ID", "BlocksetID" AS "BlocksetID" FROM "File" WHERE "BlocksetID" != -100 AND "BlocksetID" != -200
UNION
  SELECT "A"."ID" AS "ID", "B"."BlocksetID" AS "BlocksetID" FROM "File" A LEFT JOIN "Metadataset" B ON "A"."MetadataID" = "B"."ID"
)
WHERE "BlocksetID" IS NULL OR "BlocksetID" IN
  (
    SELECT DISTINCT "BlocksetID" FROM
    (
      SELECT "BlocksetID" FROM "BlocksetEntry" WHERE "BlockID" NOT IN
        (SELECT "ID" FROM "Block" WHERE "VolumeID" IN
          (SELECT "ID" FROM "RemoteVolume" WHERE "Type" = "Blocks"))
        UNION
      SELECT "BlocksetID" FROM "BlocklistHash" WHERE "Hash" NOT IN
        (SELECT "Hash" FROM "Block" WHERE "VolumeID" IN
          (SELECT "ID" FROM "RemoteVolume" WHERE "Type" = "Blocks"))
        UNION
      SELECT "A"."ID" AS "BlocksetID" FROM "Blockset" A LEFT JOIN "BlocksetEntry" B ON "A"."ID" = "B"."BlocksetID" WHERE "A"."Length" > 0 AND "B"."BlocksetID" IS NULL
    )
    WHERE "BlocksetID" NOT IN (SELECT "ID" FROM "Blockset" WHERE "Length" == 0)
  )
) GROUP BY "A"."FilesetID"
ExecuteReader: SELECT DISTINCT "B"."Timestamp", "A"."FilesetID", COUNT("A"."FileID") AS "FileCount" FROM "FilesetEntry" A, "Fileset" B WHERE "A"."FilesetID" = "B"."ID" AND "A"."FileID" IN (
SELECT DISTINCT "ID" FROM (
  SELECT "ID" AS "ID", "BlocksetID" AS "BlocksetID" FROM "File" WHERE "BlocksetID" != -100 AND "BlocksetID" != -200
UNION
  SELECT "A"."ID" AS "ID", "B"."BlocksetID" AS "BlocksetID" FROM "File" A LEFT JOIN "Metadataset" B ON "A"."MetadataID" = "B"."ID"
)
WHERE "BlocksetID" IS NULL OR "BlocksetID" IN
  (
    SELECT DISTINCT "BlocksetID" FROM
    (
      SELECT "BlocksetID" FROM "BlocksetEntry" WHERE "BlockID" NOT IN
        (SELECT "ID" FROM "Block" WHERE "VolumeID" IN
          (SELECT "ID" FROM "RemoteVolume" WHERE "Type" = "Blocks"))
        UNION
      SELECT "BlocksetID" FROM "BlocklistHash" WHERE "Hash" NOT IN
        (SELECT "Hash" FROM "Block" WHERE "VolumeID" IN
          (SELECT "ID" FROM "RemoteVolume" WHERE "Type" = "Blocks"))
        UNION
      SELECT "A"."ID" AS "BlocksetID" FROM "Blockset" A LEFT JOIN "BlocksetEntry" B ON "A"."ID" = "B"."BlocksetID" WHERE "A"."Length" > 0 AND "B"."BlocksetID" IS NULL
    )
    WHERE "BlocksetID" NOT IN (SELECT "ID" FROM "Blockset" WHERE "Length" == 0)
  )
) GROUP BY "A"."FilesetID" took 0:00:00:28.422
Starting - ExecuteReader: SELECT "Key", "Value" FROM "Configuration"
ExecuteReader: SELECT "Key", "Value" FROM "Configuration"  took 0:00:00:00.000
No broken filesets found in database, checking for missing remote files
Starting - RemoteOperationList
Backend event: List - Started:  ()
  Listing remote folder ...
Backend event: List - Completed:  ()
RemoteOperationList took 0:00:00:01.344
Starting - ExecuteReader: SELECT DISTINCT "Name", "State" FROM "Remotevolume" WHERE "Name" IN (SELECT "Name" FROM "Remotevolume" WHERE "State" IN ("Deleted", "Deleting")) AND NOT "State" IN ("Deleted", "Deleting")
ExecuteReader: SELECT DISTINCT "Name", "State" FROM "Remotevolume" WHERE "Name" IN (SELECT "Name" FROM "Remotevolume" WHERE "State" IN ("Deleted", "Deleting")) AND NOT "State" IN ("Deleted", "Deleting") took 0:00:00:00.000
Running ListBrokenFiles took 0:00:00:30.016

No remote volumes were found, refusing purge

So if I read the log-file correct: There are no missing files in the database, and also no missing files on the remote location?

So can you help me @JonMikelV (or others?)


#7

This error happens when there’s no local sqlite database (which is where Duplicati stores it’s logs). Normally when using “Direct restore” a small temporary database will be created from the files found at the destination to handle the restore process. Since no connection was made, no files were found so no database was made meaning there’s no LogData table to store the logs. I’m not sure if this is something @kenkendk has looked into yet, but even if he has I suspect it’s such a low frequency issue that it’s barely on the to-do list at this point.

It sounds to me like your destination (OD4B) might be having an issue handling Duplicati’s request for the file list when the number of archive files gets too big.

When you manually look at the destination, how many files are you seeing there?


#8

I’m having similar messages like “Found x files that are missing from the remote storage, please run repair” and "Repair cannot acquire 2319 required blocks for volume duplicati-b2fc01b08078947cd8fb26b72ae5a63d0.dblock.zip.aes, which are required by the following filesets: "

The individual files it is grumbling about do exist on the OneDrive for Business (Sharepoint) site.


#9

It sounds like the “Repair cannot acquire…” issue is a known logic loophole that apparently hasn’t been fixed - what version of Duplicati are you using?

The link above suggests that if you remove (move to another folder or delete) the “broken” dindex file then a database Repair should rebuild it correctly.

The trick is, I don’t know how to tell which dindex file goes with which dblock (or maybe @Pectojin has already shared it and I’ve forgotten) . :frowning:

If this was one of my less important backups I would try MOVING all the dindex files with the same date as the problem dblock file to another folder then run a database Rebuild which should regenerate the missing dindex files. If it worked then great, but if not I’d put all the original dindex files back. But that’s just me, and I’m a little crazy… :crazy_face: