Automatic backups stopped AND rolled back when restoring drive image of Ubuntu-partition

[Henr]

This problem is a bit complex - I think:

On Sept. the 13th I created a drive image backup of the Ubuntu partition on my Laptop. The SSD is dual-boot, so it also contains a Windows-OS-partition, an NTFS data-partition with all my personal files, and some very small hidden system partitions.

On Oct. the 6th something went seriously wrong, and I had to restore the Ubuntu partition backup.

Now I have just discovered that when logging in on http://localhost:8200/ngax/index.html#/restore/1 it says that the newest backup is from Sept. the 12th - just before making the drive image backup I restored on Oct. the 6th…

But when looking directly into the backup-‘bucket’ in Backblaze B2, the newest backup seems to be from Oct. the 5th: [duplicati-20181005T192200Z.dlist.zip.aes]…

I am almost “proud” of being able to cause such a complicated problem…
Does anyone have any good ideas about what to do?

NB! I created 3 somewhat similar topics in June-July:

"How to make Duplicati start automatically with Ubuntu?
“Automatic backups stopped on June 23rd!”
“Automatic backups not running after Ubuntu restart”

I received excellent help on all of them. Thank you!

Best regards,
Henrik R.

Important addition!:
I DO have a drive image backup of the Ubuntu partition from just before it was deleted by the restore!
I use a quite good free backup software called Paragon HD Manager 16 Basic.

[Henr]

NB: It seems to me that the basic cause of this problem is something I have experienced with several other data backup solutions:
There seems to be an index database about which files are backed up, but it’s not on the same partition as the actual files. That seems to be a logical error from my point of view.

[ts678]

I’m glad you figured it out.

I’m still trying to catch up with the “complicated problem” . My impression is it was like this:

Ubuntu runs Duplicati to do a file backup of Ubuntu to B2.

Windows runs Paragon Hard Disk Manager to do an image backup of Ubuntu to somewhere.

Ubuntu breaks.

Paragon image restores Ubuntu, including Duplicati databases used to track destination info.

Duplicati confusingly looks older than B2. Does Direct restore from backup files look correct?

Possibly running The REPAIR command might be enough to bring things back into alignment.

If repair can’t align, a stronger version of it is to (maybe slowly) recreate from the destination.

The easy way is Database management. Other harder options are safer or more informative.

If you’re willing to use the UI Comandline option, –dry-run would let you see what it proposes.

I forget whether you also need to set the –log-level, e.g. to Information, to see what it reports.

If you want recent files and don’t care about old, simply direct restore to 20181005T192200Z.

You could then start fresh by exporting the job, deleting it and destination files, then importing.

Time for fresh backup depends on how big the backup is, but at least I think B2 upload is free.

Some people elect to backup the job database right after the actual backup. For now, it’s DIY.
Databases can become quite large, and local activity can cause a lot of data to get uploaded.
Generally the desire is to speed up recovery in case of disk loss. It avoids database recreate.

Lots of options available. Which ones are best for you depends on situation and preferences.

[Pectojin]

Duplicati will keep a local database with information about your backup. It’s located in ~/.config of the user Duplicati is running under.

In any case you restored it to an old date, so you should either restore the database from your most recent backup or ask Duplicati to recreate (not repair) the local database.

[ts678]

Thanks for pointing out the location. I thought, though, that this is what was restored (stale) from the image, with no other backup copies of the database. I’m still not sure of the exact situation, but I pitched my theory on it.

It sounds like you’re pretty sure repair can’t sort this out. Ideally, it’d recreate only the new backup it hadn’t seen. Maybe one day there will be better documentation on abilities and limitations of the several Duplicati repair tools.

[Pectojin]

Honestly repair might work but I’m a bit paranoid about suggesting it in this case after this issue was reported Repair command deletes remote files for new backups · Issue #3416 · duplicati/duplicati · GitHub

[ts678]

It looks like Recreate beats Repair, not only because of paranoia (we’ll have to figure out how to either obtain a code solution or an advice solution for that – never running Repair isn’t practical, plus it’s being enhanced), but because when I tried to set up this situation by copying an old database over a current one, Repair failed:

Failed to accept new index file: duplicati-i12c3a7d045f9433c9e9ea9e9b49c62d1.dindex.zip, message: Volume duplicati-b559c362a31984483b5a7c4d7cc520c19.dblock.zip has local state Deleting
System.Exception: Volume duplicati-b559c362a31984483b5a7c4d7cc520c19.dblock.zip has local state Deleting

and beyond just tripping over itself, it also “solved” inconsistency by deleting destination to fit its own record, which is similar to your concern on #3416. --dry-run would have caught this (maybe after some deciphering).

[Henr]

Hi Pectojin and ts678!

Thank you for your suggestions!

Restoring the Duplicati database from the drive image backup made just before restoring an older drive image, thus trying to make it correspond to the current situation, seemed the smartest and easiest thing to do. So I did that. But now, when I login to Duplicati (http://localhost:8200/ngax/index.html#/log/1) I get this red error at the bottom of the screen:

" Error while running Lenovo-Ubuntu
The database disk image is malformed database disk image is malformed "

And when I press “Show”, I get this:

" Error
Failed to connect: The database disk image is malformed database disk image is malformed "

So what do I do?
The Repair command sounds a bit risky!
What about Recreate?
The problem is to get Duplicati to start doing automatic backups again EVERY day “eternally”

Clarification:
Login to Duplicati web-interface, and choosing Restore, is what shows that the last backup is from Sept. 12th.
While Backblaze shows the last backup is from Oct. 5th.

[Pectojin]

@Henr

The database disk image is malformed database disk image is malformed

Bah, this error is annoying. It’s possible it’s just an error in the database index. You can try the steps described here Database disk image is malformed - #23 by kenkendk

Alternatively I’d recreate. I’m afraid I can’t explain why the sqlite malforms as it’s not expected behavior.

[ts678]

I’m unsure of chronology and detail. Is the Duplicati database restore-from-image thought to be new enough that if fixed it would be near the latest B2 destination of 20181005T192200Z and the difference could be left to Duplicati to delete in Repair (maybe after a --dry-run sanity check) if that’s what happened? Let’s try to fix only if it may be fruitful. If not, maybe a Recreate (keep old DB copy just in case) might be better, however if you want things back faster, and possibly also dodge any latent issues in older versions, try a direct restore. Getting back to normal backup operations can then be done more leisurely. This depends on your priorities. Should your priorities include maximum data safety, a B2 snapshot is even more protection than a --dry-run.

It sounds like this image was taken just before the Oct 6 restore of Sep 12 image, so it might be worth fixing, however the conflicting result is that somehow you’re having things come up only showing Sep 12 backup…

If this was an isolated restore of the job .sqlite file, be warned that it’s not always a single file you might need to restore, but maybe related ones with different suffixes, especially if Duplicati might have been active then. See this for some types, and this commenting on suffixes. You could possibly just restore the entire folder… There’s also a question of how healthy Ubuntu was at the time. I’m not sure what “went seriously wrong” is…

I’m having a hard time reconciling the SQLite “Database disk image is malformed” error (which can also be directly looked for in a web search, in hope of finding other ideas) with the Restore being healthy enough to give a surprisingly old date, given that the image was perhaps made on Oct 6 just before restoring Sep 12.

Looking inside that database (if can be opened) with something like DB Browser for SQLite to look over the Operation table would show the chronology of operations. Timestamps are UNIX, so converters are plentiful. You can also do SQL commands for repair (if you’re an expert – I’m not) or for a “PRAGMA integrity_check;” from the “Database disk image is malformed” link above (and there are also some precautions given there).

[Henr]

OK. This seems to have solved the problem:

In Duplicati: Advanced -> Database -> Recreate

After that, automatic backups have resumed.
And it was faster and much more practical than my failed attempt to restore the database from the Paragon drive image backup.
So now everything is back to normal.
Thank you.

Henrik

[Henr]

@ts678

I’m unsure of chronology and detail.

Hi ts678

I am not really sure what you are trying to say in this last post. But my problem is solved, so thank you.