I thought I was “clever” - until now when I experienced a loss of all the backup data (the existing backups, not the source data). Does anyone have an idea if this could be connected to my non-standard backup config?
I have a backup job to an external USB drive. I moved the database AND the target data to the external drive.
Reason: I am swapping several drives - to be safe in case a drive dies / burns / gets stolen. All these drives are mapped to the same drive letter of course.
Advantage: No matter which drive I use, I just plug in any of my drives, and run the backup job. One job for all drives. In my logic, by storing the db on the external drive, the db should stay consistent to the backup data.
This worked fine for a year, but now it happened that the (incremental) backup job took much to long. I aborted and found out, that all previous backups were completely gone. Cannot tell if it existed directly before running the job, but I don’t remember deleting it.
I have a second backup job (for a different source but same system) - here the whole folder on the target drive has been deleted.
I would appreciate some ideas about:
Is this setup likely to cause trouble?
Could something bad happen, if duplicati sometimes cannot access the database (beside that the job cannot run)?
Short answer:
I’m surprised you made it this long without things blowing up and yes this will cause problems. Possible workarounds, make a backup job for each day and run each respective job on it’s day, use my powershell script, backup up to a single drive then manually copy the backup files to another drive.
Longer answer:
In spite of copying the database, the database is missing key information that’s stored in the database on the other drive from when the backup last ran to that drive and not this drive. The changes to the dblock files and database will not be reflected properly and things will eventually go wrong.
FYI: The database is strictly a speed thing you don’t need it to recover your data, the whole database can and will be rebuilt if needed from the backup data files, you can back it up but it’s actually more work to deal with an existing moved DB when it comes to restoring.
I have an external USB drive for each day of the week for all the reasons you mention. I also initially planned on using Duplicati in the same way as you did but discovered fairly early on that it wasn’t going to work as planned at which point I created the PS script (linked to above in the short answer).
So now I have a job for each day (all with the same source settings other than name) that will only backup to it’s respective drive thanks to using the alternate-destination-marker option, the jobs are then scheduled using the Task Scheduler in Windows. There are other ways to make this sort of setup work but the key is that the job and it’s database are unique to each drive/destination. The script also determines which days drives are connected and if more than one is connected will use the least recently used drive (vs the most recently used) and runs a backup to that days drive and it all shows in the GUI.
I don’t think Duplicati ever does anything like that, even if you delete a backup job and destination files.
One risk of USB drives is filesystem corruption, e.g. if the drive is pulled at a wrong time. It depends on operating system and settings though. An unplug may also damage individual files such as a database.
It’s probably safest to properly eject/unmount the drive, to be sure nothing’s active and data got written.
If all the destination files are intact, getting the database back might be as simply as doing a Recreate.
Seen where? Restore screen? Did you notice if there was anything at all, e.g. things from the job logs?
Did you save the database or note its size? Any odd error messages ow warnings while looking at job?
How so, other than maybe adding one more thing to be potentially damaged by the usual USB hazard?
The scheme was written up here with some testing, and has had some links as a possible rotation plan.