Reinstalling Duplicati on a Win10 clean installation; what db and config files do I need to copy across?

I have been successfully running Duplicati (version 2.03.3_beta_2018-04-02) on a number of computers in my household, backing up to a Linux NAS / home server. One of my computers, a Win10 PC, recently suffered a fatal corruption of the Win10 operating system and I had to restore the operating system drive from a disk image that I had created before the Duplicati installation.
Before I overwrote the operating system drive(c:) from the disk image, I saved a copy of all the directories including the Duplicati installation files.

Once I restored the disk image, I successfully re-installed Duplicati. Is there a way that I can copy over the Duplicati database and any necessary configuration files from the copy of the corrupted windows operating system drive? If possible I want to avoid the lengthy process of running a database repair.
If I can do this, can you please be specific as to the specific files and locations on the Win10 c:\ drive that I will need to copy across? I tried to search out the answer to this and haven’t been successful, hence my question.

Thanks in advance for your help!

Duplicati databases are stored in C:\Users\<username>\AppData\Local\Duplicati by default. Did you save those files before you rebuilt your system?

If so - you could shut down Duplicati and restore the Duplicati-server.sqlite file. This contains all your backup job definitions, global settings, etc.

You could also restore the other *.sqlite files, but only do that if they are fully up to date. If they aren’t, you can just ask Duplicati to recreate these other (job-specific) databases.

Thank you, @drwtsn32.

You confirmed what I thought. I do have a copy of this directory but unfortunately, copying it over to my new system drive is not working for me.

A bit more background: when I first reinstalled Duplicati and recreated the backups (without restoring the Duplicati databases from my system drive copy), I tried to run a backup and got messages about database errors so I started a Repair. It seemed to take forever so I ended up cancelling it after 24 hours as the progress bar seemed stuck at ~80%. I then posted my question here.

So, following your reply, I copied over the files in the backup of C:\Users\<username>\AppData\Local\Duplicati to my current system drive and tried to run the backup. Unfortunately, I’m still getting errors.

So I deleted the database and I am running repair again - this time I will wait it out even if it takes a week! Hopefully, this will solve my issues.

Thanks again for your help.

When you installed Duplicati on your new system, did you use Duplicati 2.0.5.1? If not, you should have. It has database recreation fixes especially for the problem where it appears the progress bar is stuck. (It’s not really stuck, but it can slow way down at that stage.)

Question out of interest here:

On my system there are multiple (physical) disks.
For example:
C:/ (OS)
D:/ (Data)
E:/ (Photos)

After reinstalling I could encounter the following:
C:/ (OS)
D:/ (Photos)
E:/ (Data)
Does this ruin the database? I would think so, right?

What I’ve done so far is reinstalling. Loading the .JSON file and run a repair/recreate.
Is this a valid option without risks or should you make sure your disks always get the same drive-letter?

It doesn’t ruin the database.

How many backup jobs do you have defined? If C:, D:, and E: folders are all in one backup job, then this will not cause much of an issue. The next time the backup runs, Duplicati will have to reprocess all the data because it will see it as new/changed files. But ultimately since the file contents are the same, no new blocks of data will have to be uploaded to the back end. The job may take a while to finish - depends on how much data you have. If you can ensure the drive letters stay the same, you can avoid this.

If the C:, D:, and E: drives are NOT in the same backup job, then you will definitely need to correct the source paths if your drive letters change. Do that before you attempt to run any backup jobs. Once you run a backup, you’ll see the same issue I mentioned above - Duplicati will need to reprocess all the data, but ultimately it won’t upload any new blocks.

I would recommend you keep the drive letters the same for simplicity, if possible.

Another reason to keep drive letters the same is because your retention policy might eventually age away old versions using the no-longer-being-backed-up path, but this may or may not matter to you.

The database uses the full paths including the drive letter, so changing drives looks like a rename, but the deduplication will figure out that all the blocks of data in the new pathname are already uploaded.

This actually might not quite be answering your question. A Recreate almost certainly only rebuilds the DB from the backup files without worrying about what’s on the drive right now, and is basically what a Repair does if DB is missing, e.g. new import of job without moving the older-but-matching-remote DB.

Tries to repair the backup. If no local database is found or the database is empty, the database is re-created with data from the storage. If the database is in place but the remote storage is corrupt, the remote storage gets repaired with local data (if available).

sounds like a Repair that is not a recreate-from-scratch might work better without a drive letter change.

Thanks for the help so far! Unfortunately, I am still struggling to get Duplicati back up and running correctly on my Win10 PC.

I am still seeking suggestions how to efficiently recover from my issues. I apologise in advance for the length of this post.

I will re-summarise the issue and what I have tried. Background first:

  1. Win10 PC backing up to Fedora based NAS/microserver
  2. Duplicati Version 2.03.3_beta_2018-04-02 (note that I intend to update to the latest version eventually but I have had problems updating versions in past so one thing at a time, please)
  3. Prefer to run Duplicati as a service and I run the tray icon with the “—no-hosted-server” parameter
  4. Three major drives on my Win10 PC:
  • c:\ operating system drive with Win10 installed
  • d:\ data drive
  • f:\ pictures drive
  1. Due to large backups, I had configured Duplicati to run two separate backup jobs, one for the data and one for the pics

As a reminder, when my Win10 operating system files became corrupted, I was able to use Linux tools (Parted Magic) to make a copy of all the files on the corrupted Win10 c:\ operating system drive. I then reinstalled my c:\ operating system drive from a disk image that I had made shortly after a Win10 clean install several years ago. At the time of the disk image, I had not yet installed Duplicati.

Objective: reinstall Duplicati version 2.03.3_beta_2018-04-02, re-create the backup jobs from saved json files and get the database repaired / sync’d to the backup data on my microserver.

Try 1: I reinstalled the Duplicati software and recreated the backup jobs from the json files. Everything fine up to this point. Tried to run the backup and I got a message saying something like (sorry, I didn’t write it down) “6925 files not found, Please run database Repair”. I went into Database option and ran the Repair option. The Repair ran for ~36 hours, the progress bar seemed to stall at ~80% and I eventually got frustrated and aborted the job.

Try 2: I figured that the files I need to get Duplicati must be present in the copy of the operating system drive with the corrupted Win10 installation. After posting here and getting suggestions, I copied across the c:\Users\<username>\AppData\Local\Duplicati directory back to my new operating system drive. When I tried to run the backup job, I received a similar error that there were missing files and I needed to run the database repair. This time, I tried the delete database and repair option. Again, the repair process was very slow and I lost the job more than 24 hours later when we suffered a minor power outage that reset the computer.

Try 3: I noticed that the Duplicati backup job screen said the database location was at c:\Program Files\Duplicati 2\data. So this time, I shut down Duplicati and tried replacing this directory with a copy from my old operating system drive. When I rebooted my computer, my Chrome browser will no longer open up the Duplicati GUI (http://127.0.0.0:8200) – I get the error This site refused to Connect, localhost refused to connect, ERR_Connection_Refused. I checked and cofirmed that Duplicati is running as a Service but interestingly, the Tray icon (–no-hosted-server) does not start up and it crashes when I try to manually restart it. I thought I had it with Try3 but I am obviously breaking the installation somehow.

I am hoping to get advice and clear instructions as to the most efficient way to get Duplicati back up and running on this machine.

Worst comes to worse, I guess, I will simply install the latest version and create brand-new full back-ups although, from experience, this takes many days. Thanks in advance for your help!

This is because you are using an old version, and didn’t do the simpler migration of copying all your sqlite files from the old computer. One solution is to upgrade to the latest version and do the recreation again. OR - delete your sqlite files on the new machine and transfer them from the old one.

I know you say you want to stay on the old version for now, but it is part of why you’re having trouble.

This may be the wrong path now that I see you are running it as a Windows service. The default location for running as a Windows service is in C:\Windows\system32\config\systemprofile\AppData\Local\. Note that this is not an ideal location to leave it as major Win10 upgrades will affect this location.

Shouldn’t it be 127.0.0.1?

There is no reason you have to start completely over and lose your backup history. Worst case, do a database recreation with the latest 2.0.5.1 beta version. But a better way may be to undo everything you’ve done and try migration again by manually copying the sqlite files to the right spots.

This person made a great post about the process:

I’ll re-emphasize that destination files and database must match, otherwise complaints will occur, and if Repair gets run, damage to the backup can occur because the backup may be fitted to an obsolete DB. Unfortunately, sqlite file names are not meaningful, but you can check dates against remote file dates…

Thank you, @ts678 and @drwtsn32! With your help, I have resolved the issues and just successfully ran my backups to prove the system is working well. I greatly appreciate the helpful community here.

To summarise what I did in case others are looking at this:

  1. Upgraded to Version 2.0.5.1_beta_2020-01-18; despite my bad experience updating versions in the past, this upgrade turned out to be painless. I really liked the improvements to the logs.
  2. Stopped the Duplicati Service
  3. Copied all data in my backed up copy of c:\Users\<username>\AppData\Local\Duplicati drive over to the same place on my new Win10 system drive
  4. Copied all data (database info) from the backed up c:\Program Files\Duplicati 2\data to the C:\Windows\system32\config\systemprofile\AppData\Local\ area on my new system drive. Based on drwtsn’s advice, I needed the database files in this area since I run Win10 as a Windows service.
  5. Restarted the Duplicati Service and opened up the GUI (yes, I had a typo, it is at (http://127.0.0.1:8200. Backup jobs were present and I successfully ran then to confirm that everything was working.

@drwtsn32, you say

The default location for running as a Windows service is in C:\Windows\system32\config\systemprofile\AppData\Local\ . Note that this is not an ideal location to leave it as major Win10 upgrades will affect this location.

How do I configure a new location for this databse information and where would you recommend that I place it so that it is not interfered with by a major Win10 upgrade?

Thanks!
Carl

Glad you got it working!

You can use the --server-datafolder option to change the location of Duplicati-server.sqlite file. Shut down Duplicati and append that command line option to the service. For example:

--server-datafolder=C:\ProgramData\Duplicati

Alternatively you can set the Duplicati_HOME environment variable if that is easier:

Duplicati_HOME=C:\ProgramData\Duplicati

Move the Duplicati-server.sqlite file to this new location and start Duplicati.

You will also want to move the job-specific sqlite files. To do that move the file and then edit the database path for that job in the Duplicati Web UI. (Click the job, click Database…, change the path and click save.) There is a Move existing database button - maybe you can use that instead of moving the file yourself. I’m not sure, I’ve never tried it.

I used the move database button today and it works without problems.

1 Like