Newbie: Crash Durring Initial Backup. Repair, Restore fail: No Filelist found

Support
1

First off, thanks for the great software.

I’m running 2.0.4.5_beta_2018-11-28 on Arch Linux, attempting an AWS S3 backup. This is my first time using Duplicati (or any real backup solution, outside of some simple rsync), and I’m not entirely confident I understand how it all works.

My Xorg server has been crashing lately, and it seems to have crashed about 100G into my 300G initial backup. I can, of course, begin the upload again, but it begins again (the full 300G) instead of resuming where it left off (at about 200G remaining).

There are still about 300 zipped files in my AWS bucket, as seen through the AWS web interface.

What I’ve tried, followed by the error message:
Attempt to resume: The database was attempted repaired, but the repair did not complete. This database may be incomplete and the backup process cannot continue. You may delete the local database and attempt to repair it again.

Attempt to Repair : No filelists found on remote destination.

Attempt to Recreate: No filelists found on the remote destination.

I’ve also tried the above on the command line, to no avail.

At some point, I was also getting “Found 1919 remote files that are not recorded in local storage, please run repair”. I forget what I did to get to that point, but I haven’t been able to recreate it. (my guess is I was deleting the local database and trying to backup before repairing? I’).

Stored Server logs from when (I assume) it crashed:

Live Logs:

  • Dec 28, 2018 12:37 PM: The operation Repair has failed with error: No filelists found on the remote destination

  • Dec 28, 2018 12:36 PM: The operation Repair has failed with error: No filelists found on the remote destination

  • Dec 28, 2018 12:36 PM: The operation Repair has failed with error: No filelists found on the remote destination

  • Dec 28, 2018 12:35 PM: The operation Backup has failed with error: The database was attempted repaired, but the repair did not complete. This database may be incomplete and the backup process cannot continue. You may delete the local database and attempt to repair it again.

  • Dec 28, 2018 12:28 PM: The operation Repair has failed with error: No filelists found on the remote destination

  • Dec 28, 2018 12:28 PM: The operation Repair has failed with error: No filelists found on the remote destination

  • Dec 28, 2018 12:27 PM: The operation Repair has failed with error: No filelists found on the remote destination

  • Dec 28, 2018 12:05 PM: The operation Repair has failed with error: No filelists found on the remote destination

  • Dec 28, 2018 12:05 PM: The operation Repair has failed with error: No filelists found on the remote destination

  • Dec 28, 2018 12:05 PM: The operation Repair has failed with error: No filelists found on the remote destination

  • Dec 28, 2018 12:04 PM: The operation Repair has failed with error: No filelists found on the remote destination

  • Dec 28, 2018 12:04 PM: The operation Repair has failed with error: No filelists found on the remote destination

  • Dec 28, 2018 12:04 PM: The operation Repair has failed with error: No filelists found on the remote destination

  • Dec 28, 2018 12:00 PM: The operation Repair has failed with error: No filelists found on the remote destination

  • Dec 28, 2018 12:00 PM: The operation Test has failed with error: Found 1919 remote files that are not recorded in local storage, please run repair

  • Dec 28, 2018 12:00 PM: Found 1919 remote files that are not recorded in local storage, please run repair

  • Dec 28, 2018 11:49 AM: The operation Repair has failed with error: No filelists found on the remote destination

  • Dec 28, 2018 11:48 AM: The operation Test has failed with error: Found 1919 remote files that are not recorded in local storage, please run repair

  • Dec 28, 2018 11:48 AM: Found 1919 remote files that are not recorded in local storage, please run repair

Thanks for any help.

2

Hi @funnyguy, welcome to the forum!

You’ve got quite a little pile of issues there, but we can work through them. :slight_smile:

This is most likely just how it appeared, not what was actually happening. When a backup starts it normally scans your entire file list causing the progress bar to show your full number / size of files “to be done”.

But Duplicati will check it’s local database and if a file has already been backed up (no change in date/timestamp) it will quickly move on the the next file causing the progress bar to “count down” as if it were actually re-backing up the files - but it’s not. Even if a file has a timestamp change, Duplicati will break it up into blocks (100k by default) and only back up the blocks that have changed from the last time.

There’s a bit of a delicate time during the initial backup where attempting a Database Repair (or Recreate) actually makes things worse. (@kenkendk, is there some way to avoid this issue if we don’t have at least 1 complete backup?)

As I understand it, this is because the repair (and recreate) process wants to use dlist files from the destination, but those files aren’t uploaded until the backup is complete.

This is likely because your local database was gone (due to an attempted recreate) so Duplicati created a new database (as it should) then looked at the destination and said “whoops - there’s a bunch of files out there I don’t know about”. Normally a Repair would resolve this, but without the dlist files (because you didn’t have 1 complete backup yet) the reapir won’t work.

If you want to try and keep this backup, I’d suggest looking in your Duplicati folder (I think it’s something like ~/.config/Duplicati or /.config/Duplicati) for a “backup 2018mmddhhmmss.sqlite” file that Duplicati should have been made prior to the attempted database Recreate.

Stop Duplicati and copy that file back to it’s old file name (something like CTRMXALMWD.sqlite - whatever shows in the “Local database path” of the job Database menu) then start Duplicati again.

This should get you to where you were before the attempted Recreate. From there, try starting the backup again and letting it try and finish - even if it looks like it’s starting from scratch (even though it isn’t). :slight_smile:

Alternatively - if you don’t want to go through the above hassle, declare this backup a test run, delete it (and the destination files - sorry) and start over, but this time I’d suggest you create your job with a small set of source files.

That should run pretty quickly and then you can do a test restore to make sure you understand the full cycle of backing up AND restoring. Once you’re happy with that, edit the job to include more (or even all) of your total files to be backed up.

Now if your Xorg server crashes while running the big backup the Repair process should work better because it’s got at least one dlist file to work from. :smiley:

3

Thanks for the help. I attempted to restore the oldest backup.sqlite file; however, it didn’t work (I got the unexpected remote files error).

However, the timestamp on the oldest backup is at 11:34… well after the error logs show errors.

I’m going to call this a test run.

I’d like to actually start my next (smaller) upload with a few subdirectories. I’m guessing the best way to do to this would to be to recreate the source file path under “Backup Destination: Folder Path” settings.

For example, my goal will be to backup all of home/ . However, to avoid this issue in the future, I’ll start by backing up subdirectories /home/a , then home/b/1/ ,then home/b/2/, and then, finally, just plain-ol’ home/.

My assumptions about the best way to do this:
Under “Backup Destination”, set “Folder Path” to the correct path of the subdirectory I want to backup (relative to home/): a/
Obviously, set the source dir to home/a/.

After that backs up, switch it to Folder Path to b/1/, and change the source dir again.

Then, eventually, just back up all of home/.

I’m assuming that dupliciti will not complain between each of these steps that there are unknown files in the system since… well, it put them there. But I’m not sure how it handles changes to destination (i.e. Folder) and Source paths.

Thanks

EDIT: rather than do all that, I just selected a small text file in /home as well as one subdirectory for my first backup. That way, the correct file structure is created (I assume).

4

Good choice. Your previous plan would have worked except REPLACING the source path with have caused all the 1st path files to be considered deleted.

The way I do it is to start with a small deep folder then go “up” such that the previous folders are included inside the new one.

And of course you can have more than one source folder. :slight_smile: