Restore Problem

1

I own 2 PCs - PC1 and PC2.

On PC1 Duplicati backup runs once a day at 17:00 o´clock in the afternoon.
Sometimes a program still occupies some files. This causes backup to fail.
Closing this program and running Duplicati backup again (“Run now”) repairs the problem and reports a successful backup.

The purpose of PC2 is to restore backups of PC1 in case of a failure of PC1. The configuration of Duplicati on PC2 is an import of the Duplicati configuration of PC1.
Of course I have to Recreate (delete and repair) local database on PC2 every time before restoring files.

After a backup fail on PC1 and following successful backup it is however not possible to Recreate database on PC2.

Error log on PC2 shows “Remote file referenced as duplicati-x…x.dblock.zip.aes by duplicati-y…y.dindex.zip.aes, but not found in list, registering a missing remote file”.

Restore of files is not possible in this case.

What can I do to Recreate PC2 database and restore files to PC2?

2

In the meantime I found this in Duplicati Manual:

In disaster recovery scenarios, the Duplicati Recovery Tool performs 3 steps:
• All remote files are downloaded from the backend, decrypted and stored in the local filesystem.
• An index is built that allows Duplicati to keep track of what information is stored in which file.
• Files are restored from the downloaded backend files by recreating them using the blocks inside the .DBLOCK files.

Unfortunately this recipe fails already in step 1.

The reason for failure is simple:
Step 1 requires the entry of a passphrase similar to this --passphrase=“4u7P_re5&+Gb>6NO{”

My passphrase was generated by Duplicati and contains a "-sign in the middle. This prevents successful decryption of encrypted files.

Now I´m looking for a way to enter passphrase despite of this “xxxxxxxx**”**xxxxxxx" problem.

3

I’m not sure the RecoveryTool is the right road to go down. If PC1 backups are working and it’s not complaining about files on the back end, then you should be able to recreate the database on PC2 which is pointed at that same destination.

It is important to make sure Duplicati on PC1 isn’t doing anything while the recreation is in process on PC2. Otherwise it will confuse PC2.

By the way, to answer your question about the password with the command line tools, it should work to quote the string. Dashes shouldn’t be a problem but if your password has a “%” sign in it you may need to escape it as cmd interprets it as variable expansion.

4

I tried to use RecoveryTool, since Recreate database in GUI-version on PC2 is definitely not possible.
I get this error message:

2021-09-02 17:58:03 +02 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-MissingFileDetected]: Remote file referenced as duplicati-b31678fb34b1340eb8c71b0319e6e36c2.dblock.zip.aes by duplicati-i14da8949ce32484a8f879a69f445d17d.dindex.zip.aes, but not found in list, registering a missing remote file

PC1 isn´t doing anything because it is switched off when I try to recover files on PC2.

The password does not contain a %-sign but it contains a "-sign.
Example password

asdf"yxcv

When I use command --password=“asdf"yxcv” probably only the first part of the password seems to be used resulting in a password error.

Of course I´d prefer to use Duplicati GUI, but I didn´t find a way to overcome this error mesaage above.

5

Yeah, recreate failing on PC2 is a problem. I bet if you turned on PC1, it would complain about your back end files. Can you test?

Oh I see, I thought you meant you were using a dash but now I see you meant you were using a double quote. Try putting a backslash in front of the quote and see if that helps:

--password="asdf\"yxcv"

But really I’d not go down this RecoveryTool road just yet. Try running another backup with PC1, I bet it will complain about your back end files.

6

Duplicati backup on PC1 reports “Success”.
I got latest success e-mail message just minutes before above error message.

From earlier attempts I learned, that I can solve the problem by completely deleting all files in the backup folder (stored in Microsoft OneDrive) and then creating a new backup.
But doing so I loose all older version of the backup of course.

7

You shouldn’t need to start completely over. That’s quite drastic and not needed unless something happens to your backup files that makes them unusable.

I’m kinda stumped then. I don’t know why PC1 would have no problems with those back end files, but then PC2 would indicate there is a problem with them when you’re doing a recreate.

Maybe someone else has an idea.

1 Like
8

Thank you, I appreciate your thoughts very much.

9

First, some problem diagnosis is required, but before that, how did “backup fail on PC1”?

I suspect you might just have an extra dindex file, but proving it’s extra is kind of difficult…
If you like, you could rely on the error message plus perhaps at least some sanity checks.

Sanity check:

If you have more files at destination with dindex name than dblock, you might have extra.

If the dindex file listed in the error has a timestamp near end of failed backup. that’s a clue,
however if timestamp is earlier the problem may have been seeded earlier. Got good logs?

Better check:

Creating a bug report from PC1 and posting a link to it so somebody can look at database.

Harder check:

If you are willing to copy the database and look in it, you can install DB Browser for SQLite.
Database browsing is similar to looking at a spreadsheet. I can provide more detailed info.

Basically you would look for the mentioned dindex in Remotevolume table Name column,
then look for that ID in IndexBlockLink table.IndexVolumeID column. It’s probably not there.

If not there, you can just delete it from the destination. If in doubt, you can move it instead.
The purpose of a dindex is to index a dblock (whose name it knows). You seem to have a
dindex that has no dblock, so it’s not doing anything useful, and is generating a complaint.

Remote file referenced … but not found in list, registering a missing remote file #4586
is what you might have hit, or maybe the cause was different but the damage was similar.

The problem I had is latent and backup runs fine until a database recreation notices issue.
This means it could also be rather old. The time on the dindex file might say when it arose.

10

Thank you for your answer. This is the error message on PC1, when problem started:

Failed: Found 9 files that are missing from the remote storage, please run repair
Details: Duplicati.Library.Interface.UserInformationException: Found 9 files that are missing from the remote storage, please run repair
   bei Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList(BackendManager backend, Options options, LocalDatabase database, IBackendWriter log, IEnumerable`1 protectedFiles)
   bei Duplicati.Library.Main.Operation.BackupHandler.PreBackupVerify(BackendManager backend, String protectedfile)
   bei Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__20.MoveNext()
--- Ende der Stapelüberwachung vom vorhergehenden Ort, an dem die Ausnahme ausgelöst wurde ---
   bei System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   bei CoCoL.ChannelExtensions.WaitForTaskOrThrow(Task task)
   bei Duplicati.Library.Main.Controller.<>c__DisplayClass14_0.<Backup>b__0(BackupResults result)
   bei Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action`1 method)

Log data:
2021-08-30 18:19:32 +02 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingFile]: Missing file: duplicati-20210826T150422Z.dlist.zip.aes
2021-08-30 18:19:32 +02 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingFile]: Missing file: duplicati-b6847faf6c5714bfaa08f2974db951ed5.dblock.zip.aes
2021-08-30 18:19:32 +02 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingFile]: Missing file: duplicati-if3caf00cfb444779b1ae92d356aa04d7.dindex.zip.aes
2021-08-30 18:19:32 +02 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingFile]: Missing file: duplicati-20210830T042242Z.dlist.zip.aes
2021-08-30 18:19:32 +02 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingFile]: Missing file: duplicati-bc1f1bd6587c94f2b8d3a1f8fd36ce26a.dblock.zip.aes
2021-08-30 18:19:32 +02 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingFile]: Missing file: duplicati-i05acebf621264500a21f76137d73262a.dindex.zip.aes
2021-08-30 18:19:32 +02 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingFile]: Missing file: duplicati-20210830T150400Z.dlist.zip.aes
2021-08-30 18:19:32 +02 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingFile]: Missing file: duplicati-b2063c95de3714f4683964d7c105b5a67.dblock.zip.aes
2021-08-30 18:19:32 +02 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingFile]: Missing file: duplicati-i59edfae8342849ceb084298964a9ab94.dindex.zip.aes
2021-08-30 18:19:32 +02 - [Error-Duplicati.Library.Main.Operation.FilelistProcessor-MissingRemoteFiles]: Found 9 files that are missing from the remote storage, please run repair
2021-08-30 18:19:32 +02 - [Error-Duplicati.Library.Main.Operation.BackupHandler-FatalError]: Fatal error
Duplicati.Library.Interface.UserInformationException: Found 9 files that are missing from the remote storage, please run repair
   bei Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList(BackendManager backend, Options options, LocalDatabase database, IBackendWriter log, IEnumerable`1 protectedFiles)
   bei Duplicati.Library.Main.Operation.BackupHandler.PreBackupVerify(BackendManager backend, String protectedfile)
   bei Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__20.MoveNext()

After this error occuring on PC1 I tried to “Repair” and “Recreate” Duplicati database on PC1 - both were not successful.
Then I started a new backup, which was successful. On PC1 everything seems to be fine - I´m however not shure, whether I can rely on this.

This is why I tried to restore data on PC2, which led to the described problem.

I can try to look into Duplicati database, but I´m afraid not to have sufficient knowledge to get some benefit from that.

By the way, I started using Duplicati appr. 9 month ago and this is the 3rd same problem since then. It would be good to know , what causes this problem in order to avoid this in the future.

Edit:
I now had a look into Duplicati database of PC1. This database has a size of 180 MByte.
As I suspected - I don´t understand anything.

11

PreBackupVerify finding an inconsistency likely means problem was introduced by prior backup failure, which unfortunately does not always leave good logs. Because you have a mostly-scheduled backup set (mostly because I see Missing file: duplicati-20210830T042242Z.dlist.zip.aes which isn’t the 17:00 run, such as duplicati-20210830T150400Z.dlist.zip.aes – 17:04 in local time of UTC+2) the logs may have some gaps which show failures. Check logs in both the usual spot under the backup, and About → Show log → Stored which is the server log, which tends to get the error if backup doesn’t finish.

Any recollection of how they failed, especially PC1 Recreate? For example, did it fail like PC2 Recreate?

Any Recreate works from the destination files (what is your destination BTW?), so should fail the same.

You might have a destination file problem, which is why I gave a Sanity check and Better check that could be considered. You don’t have to head directly into the Harder check, because it’s proving hard…

12

Thank you for this analysis.

Besides the scheduled backup at 17:04 I created a duplicati.cmd file, which is envoked when the computer is shut down. All other programs are shut down when Duplicati starts. This might explain the different times of backups.

To me the error messages seemed to be pretty the same on PC1 and PC2. But as there is a lot of text which I do not understand in detail, I did not examine this in detail. Duplicati sends a mail to me after backup, the error messages are copied from these mails.

PC1 is at a remote place some miles away, PC2 is local. I can access PC1 from PC2 via TeamViewer.

Tomorrow I´ll be at the location of PC1. I will delete these faulty backup files completely and run a new backup (appr. 12 GB backup size). The backup is stored on Microsoft OneDrive. Connection is via FTTH at a speed of 100 MBit/s and this is very stable (line capacity is 1GBit/s). That´s why I don´t think that I have a destination file problem.

I hoped it would be easier to find the reason for the problem, so that I can avoid it in the future.

Thank you for your help.

13

You are triggering a backup on shutdown? This is probably a bad idea. Windows only gives programs some number of seconds to close before they are forcibly terminated. Forcibly terminating Duplicati during an operation can sometimes lead to problems with your database or back end files.

14

I might misunderstand, but the fact that you plan to delete all destination files says you think you have a destination file problem. In addition, pretty much by definition, a Recreate failure says something’s bad.

Having a stable line is nice, but doesn’t prevent all problems, especially if Duplicati is killed at shutdown.

15

If you have a look at gpedit.msc > Computer Configuration > Windows Settings > Scripts you can add scripts. Shutdown is done after finishing these scripts. Thus Duplicati is not killed at shutdown. My script is generated from Duplicati Configuration Export As Command-line copied to a *.cmd file. If I add a pause command at the end of this *.cmd file, the computer never turns off.

This solved my problem already 2 or 3 times before. Only by deleting all destination files and executing a new backup (this is done on PC1) solved my problem on PC2. After that I could Recreate database on PC2. Without this procedure I did not manage to Recreate database on PC2 without error.

Btw. Recreate works on PC1 before deleting destination files, but not on PC2.

16

Was PC2 tested at exactly the same time, or at least with no backups or other work in between?
I can’t imagine why two systems would give different Recreate unless the destination files varied.

You also said:

which (though it wasn’t sure) is what I’d expect if Recreate and nothing else was done on both systems.

This by itself is possible and suggests a destination file problem, but PC1 should get the same error then.
There have been a variety of errors mentioned, but I’ll go back to original post which talked only about the

and if that’s the only PC2 Recreate error now (is it?), I suggest again that you test deleting named dindex file to see if you can manage to Recreate database on PC2 without error. You don’t HAVE to, but did ask:

and that’s my proposed answer to a problem that only involves that specific message and no other ones.
It would have been nice to have some of the other suggested tests, but before deleting it all, why not test this one and avoid losing all your history like you did before? You can still delete it all later, if you prefer to.

17

Very cool. I knew about that GPO setting but didn’t realize it would wait indefinitely for the scripts. I learned something new!

18

By chance I found out something rather strange:

Up to now I had set Backup retention to Custom backup retention 7D:U,4W:1W,36M:1M on PC1 and also on PC2.

I changed this to Keep all backups on PC2 only.
Now database Repair on PC2 worked without any problem.
So I do not have to perform this delete procedure.

I´m happy (for the moment)

19

You will be unhappy again. Just learn the following:

a) There’s always some (complex) problem appearing on restore, especially with fresh computers. Beware of the fresh computer.
b) Plan to spend some days in Fora.
c) Hence the restore can, and this is just the best case, only be done by yourself.

I had the idea to backup for other people, let’s say in case I leave this world. This is not realistic. No average person, not even IT people, would be able to manage the restore. Realistically, your data will just be forgotten.
Consider a drag+drop 7z solution for relatives. I did that. Drop a folder on a cmd script and get any file encrypted one by one to single encrypted files, a for loop. Decryption as easy. “To single files” is important, because of corruption and loss.

20

This is a problem with any backup. If you don´t tell your relatives, that you have put some money inside your old couch or sofa, they will throw away this old piece of furnitur without pulling out the money before.

But the purpose of Duplicati backup is different. It´s purpose is to limit the amount of loss of the results of my daily work just in case of a hardware failure.

And of course (1) Duplicati is not the only backup I have.
Every 2 weeks and inbetween after a major modification of the system (program installation or Windows updates) I run a backup of the whole system with a different backup program to an external hard disk drive, which is disconnected and stored away after backup.

And of course (2) I write an electronic documentation how this backup is done and can be retrieved and I have a printed (on paper) documentation, which describes how to get access to that electronic docs.

Btw.: did you tell your relatives the passphrase of your encryption? In a written form?