Hi, I have an 11Gb file /root/.config/Duplicati/HODCIXUAER.sqlite which is making me think twice about placing /root/.config/Duplicati into the backup itself. (I know, very meta.)
How do people usually go about saving their duplicati configuration, so that they can easily restore it later?
Kind regards,
When I had been using Duplicati for a short while, my computer crashed and I had to do a restore without any metadata. After that bad experience I decided to back up the job databases in a separate backup job.
Recently, one of my computers again crashed and I needed to perform a restore. This time, I was much more familiar with Duplicati, and knew the procedure much better. I learned that all I need is to re-create job definitions, either from a saved .json file, or just by typing in the info in the web ui. I didn’t have .json files, but I knew the job names, so I manually created my backup jobs anew, not even having the full exclude lists at hand. This I wasn’t able to do the right way back in the early days, but this time I knew exactly what to do.
Once I had done the restores I needed, I then updated the new job definitions and saved them into .json files with the export functionality. I created a separate directory for them, ~/.config/Duplicati-json, which I include in my backups, or at least in one of them. I cannot be 100% sure I remember to do the export into .json files every time I modify the job definitions, but after the latest experience, I know that the most important information to have at hands is passwords and storage locations. Once you have created some way or another a job definition that points to a storage location, you can then view what is in there and update the job’s source list and filters accordingly.
I have now dropped the separate job that backed up my ~/.config/Duplicati, as with the latest experience, I know I don’t need it.
Great answer IMO from @AimoE. Others have gone down the same path of relying on database recreate rather than backup. It probably does depend on whether you can tolerate a worst-case recreate or a failure.
Preserving configuration for me is an occasional Export To File saved various places including remote one.
was a post from me yesterday, and there’s lots more there because someone asked about good practices, prompting a list, then a later note that what’s good for a person may vary with needs and actual experience.
There’s also a point about database recreation testing, including DB rename rather than just-hit-Recreate. Yesterday I tried a rename-Repair, and my DB got much smaller, and I got a test instead of just a vacuum.
Now that you say that, I realize I didn’t even perform a recreate, not explicitly. I don’t even know if that has occurred somewhere along the way, though. Funny. All I did was to restore the files I had not been able to restore from Windows File History, and then I patched up the job definitions, and then started running backups again.
Thanks for reminding. I hope it won’t get me into trouble.
A long time ago I used to back up the large sqlite files to avoid a bug where Duplicati might take a very long time to recreate the databases. I haven’t had to do that in a while, though, as I believe the bug has been fixed.
In my understanding the bug caused dindex files to sometimes be written incorrectly on the back end, and during a recreation this incorrectness was detected and forced Duplicati to download dblock files. (When that happens the recreation will take much, much longer.) Note that using the latest beta of Duplicati doesn’t automatically fix dindex files that were written incorrectly by the buggy version, but there is a way to have Duplicati re-write the files.
What I do today is keep json exports of all my backup jobs, and also copy any global options I set into a text file. This allows me to reconfigure Duplicati quite easily in the case of a disaster. Database recreations don’t take very long in my testing, but it does depend on how large your backup is.
Reading through these responses, it occurred to me that we should be able to (from the command line) ask duplicate to dump our JSON configuration file (with/without passwords). Then we can place it in a cron job.
This tool may make that easier: GitHub - Pectojin/duplicati-client: A command line client for controlling the Duplicati Server
It talks to the web service version of Duplicati (where your jobs are defined, unless you’re using Duplicati purely from command line). It does have an “export” function but I haven’t tried it myself. I have used this tool to trigger backup jobs though, it works great.
Command line is completely independent from the server, and the server is what covers config files.
Speaking more loosely of command line (not specifically Duplicati.CommandLine.exe), perhaps this
Add ability to import backup configurations from command-line #3595 might be the basis of exporter.
Volunteers are always less than wishes, but for any coders, a standalone might be an easier start…
I forgot to mention that the job configs for the server are in Duplicati-server.sqlite. You can copy that.
If you like, encrypt it with SharpAESCrypt.exe to protect sensitive info, or store it somewhere private.
--control-files (Path): Set control files
Use this option to attach extra files to the newly uploaded filelists.
might be another way to upload it embedded inside a dlist zip file (hopefully encrypted for safety).
Getting file back would need a decrypt and unzip. You’re probably adding some space, but not a lot.
On restore, a stale file may mean stale statistics, “missed” job run, and potentially old configuration.
Stale regular Export is a bit less disruptive and gives you control over solving issues at job creation.
Whatever you do, test to see if you like it.
If you mean that the job definitions should be included in the backed up data, then yes you can back up the Duplicati-server.sqlite or you can back up individually exported .json files. But if you mean that you would want the contents ofa job definition to be deduced from the old backups, then I cannot see how it would be reasonable. The job definition includes the encryption password as well as the storage’s user account and password. Do you really want that kind of information to be deductible to anyone? The most important part of a job definition is the combination of sources and filters. But those both may have changed over time, and even the latest might not be valid on the new machine that replaces a crashed one. Plus I definitely do not want to have the old schdule activate the jobs when I restore the job definitions.
To me, the only rational way is to keep a backup of the exported json files, and edit them to disable scheduling, before loading them to the web ui (and thereby into a fresh new Duplicati-server.sqlite).
I took the “should be able” to to be a request for command line exporter (maybe we’ll hear answer).
Kind of vague. Destination doesn’t have that, nor does the job database. It’s in the server database.
One security question is whether command line (less protected than GUI) should hand out the info.
Skilled attack on the system could attack the database directly, but should an easy-read be added?
A command line utility to export configurations shouldn’t be too difficult, although there are likely some nuanced details to consider. For now, I think duplicati-client provides the ability to export configurations from the command line.
EDIT: I just noticed that @drwtsn32 already mentioned duplicati-client above. 