I have a 330 GB directory on my PC that backs up to Dropbox every hour. It has been working without issue for about 2 years now, and I haven’t touched any settings for months.
A few days ago I got the error:
The database disk image is malformed database disk image is malformed
So I recreated (delete and repair) the database, which took about 18 hours to complete. That fixed the problem for about 24 hours, but I just got the error again.
None of the other posts about this particular error on this forum have provided a working solution. Memtest suggests my RAM is fine, and to the best of my knowledge there is nothing wrong with the SSD the directory is on (e.g., I have a number of large Rdata files on here that are regularly modified, and all of them still have their integrity).
Running Duplicati - 2.0.6.3_beta_2021-06-17 on Linux.
How To Corrupt An SQLite Database File gives a pretty long list of things that might be causing this problem which is below the Duplicati level. Duplicati may mess up the database contents at the database table level, but I don’t know of a way it can obliterate the database at a low level to the the point of it not being opened…
Have you tested seeing if it’s openable at the lowest level, maybe even try cat it into /dev/null or something? The message sounds like SQLite could open it enough to decide that the format was too wrong to continue.
Is this finished-fine-resumed-broken all on the same Duplicati run, or are there restarts of any kind between? SQLIte ideally is immune to all sorts of abuse (including sudden kills), but the cited link shows vulnerabilities.
There’s a report somewhere of a Linux system that had a lot of these. Duplicati uses the distro SQLite, and potentially these can be old, e.g. if distro is an LTS that seeks low change. Any idea where yours stacks up?
https://repology.org/project/sqlite/versions shows what various distros ship. Maybe you have an old version?
This would not explain the timing of the sudden problem after 2 years of success. Any recent change done?
EDIT:
As a long-shot, do you backup the Duplicati Database for the job being run? There’s not much point, as it’s instantly obsolete, and some people suspect that Duplicati can lock itself up due to doing self-interference. Your logs don’t look that way. What I see looks like a nice backup, but later (after what?), DB is malformed.
I think I remember someone else having this issue because of a low disk space issue. Double check the free space in the filesystem where the sqlite database is stored. If you have a 1GB database (for instance), make sure you have at least that much space free. (During a vacuum operation a new database file is generated from the first, so during that operation the database will take “twice” as much space.)
No restarts of the PC or of Duplicati. The system was sleeping (suspended) for about 10 hours overnight, but there were multiple successful hourly backups both before and after this.
Just the the standard version that’s rolled out with the latest Linux Mint 20.3 and Ubuntu 20.04. libsqlite 3.31.1-4ubuntu0.2. The repology page does say that 3.31.1 is an outdated version. I don’t think there have been any changes to this in the last few weeks.
I did have a few problems with a kernel update two weeks ago (e.g., it broke cryptsetup-initramfs for some reason, which prevented the system from booting properly). But that was a couple of weeks ago, and this has only been a few days.
No, I’ve explicitly excluded the Duplicati database from all my automated backups (Back In Time etc.). It’s a large file with constant changes, and I couldn’t see any point in it.
This is what I don’t understand. One hour it is working fine, the next it is malformed. Literally the only thing I was doing at the time was writing in LibreOffice. And it has now done this twice.
It is making me question whether perhaps my SSD is failing. But I’m a heavy user, and I haven’t noticed any other problems whatsoever.
I’ll definitely give this a shot and go through the list from the link you provided when I have some time to properly troubleshoot.
Just an update on this. The problem has resolved itself, which is strange.
After a week not touching it, I just sat down to look into it again. I checked the database file to see if it had been modified since I last ran Duplicati, thinking something else might be interfering with it. But it hadn’t.
So I was about to try ts678’s suggestions for troubleshooting around SQLite, and thought I’d try running it again first, just for the heck of it. And it worked. It completed the whole backup as normal.
I have no idea why.
I’ll post an update here if the problem occurs again, or in a few weeks if it’s still running without problems.
