Backup Reconnect

Support
1

(I’m not quite sure of the best category for this topic is since it crosses feature requests / support / installation / etc… but it is first and foremost a support question. Feel free to move it as needed.)

I recently had a motherboard failure and had to build an entirely new system, CPU / RAM / etc… and it looks like the approach i took to reinstalling duplicati wasn’t a good one.

After system failure, none of my data was lost. I lost access to the backup configuration files (because i couldnt find it before i formatted the os), but all of my local hard-drive data was valid. I actually did manage to preserve the sqllite databases but that didn’t really end up helping me at all. I also didn’t know the version number of the previous install but it was definitley some 2.0.* release at least 1-2 years old (though updates were enabled maybe?)

I did some backups of local filesystems, and checking of remote filesystems, and wanted to make sure the new version worked properly with the older data. I was fully able to restore from my backups on dropbox and that worked great.

However, when i tried to setup a backup to the same folder in dropbox and the same local folder, duplicati deleted everything and uploaded a new copy of everything. I’m pretty sure my configured chunk size was different, so i’m not entirely surprised that it didn’t work, but just sliently wiping out the remote backup seems pretty rude.

I had minor local modifications in my local copy compared to the remote copy so this could have contributed but it seems unlikely.

After one backup being completely deleted (i had a copy so nothing was actually lost). I decided i should just fork and start a new backup set in a different folder. But from a longevity standpoint it seems like duplicati team needs to come up with a solution to this.

Is there actually a way to handle this use case that i didn’t find or execute properly? I looked through the documentation and didn’t see anything (though i did see somewhere that configuration mismatches would probably be problematic).

There are probably a couple use cases worth considering like, where are the master configuration files stored on my debian system (and of course other operating systems) so i can grab them next time? Is there any point in keeping the old sqllite databases and how would you get the web interface to use them? What is the proper procedure for reconnecting to a backup that was disconnected due to software reinstall / operating system reinstall / etc… given some / all / none of the different pieces of data that would be useful in this process?

Thanks in advance and i really appreciate all the hard work that has gone into Duplicati!

2

Those files actually should have helped you, as Duplicati2 is able to use them, so the real question here is what exactly happened, what did you do with these files and how exactly did things go wrong?

3

I didn’t do anythig with those files and duplicati didn’t seem to offer me any option.

  • I entered the webui
  • checked import but it doesn’t offer to let me use the sqllite db (ERROR: Unexpected character encountered while parsing value: S. Path ‘’, line 0, position 0.)
  • setup a fresh backup configuration that points to the same folders
  • clicked run
4

I hope someone who has been there will be able to help you. While waiting, I can open up the procedure a little bit for you.

I have lost my system disk and had to do restore without the local databases, because they were not backed up - and then after having restored the latest versions, I decided to create a new arrangement of backup jobs, so I didn’t continue using the local databases that the restore operation created for me.

Anyway, guessing from your description, it might be possible (and this is just guessing) that the location of the local databases may have been changed between versions. I don’t know for sure, and it is not something you could be expected to know about all of the sudden. The Duplicati manual pages are not version-specific and digging up notifications of the past releases is not something one can expect you to do when you are in the situation that you are.

Note that there are two kinds of local databases. There is a file named Duplicati-server.sqlite, which holds the backup job specifications and some more info about their general status. And then there are job-specific databases, which record the state of the backup block files at the target location, and some more information. When I ran the restore operation, it re-created the job-specific databases for me, but in order to use them, I would have needed to add a specification for each job, and while doing, it, manually point each job to the existing job-specific database. In other words, I would have needed manually re-create the server database, but only that one.

As I said, I have never done that myself, so this is just to give you an overview.

EDIT: By “manually” I mean via the Duplicati web UI.

5

Discussion in another topic reminded me of a natural reason why you might not see the jobs: you may have been running Duplicati in different mode. For example, if you ran Duplicati in service mode before the crash and now you tried the reconnect in user mode? Or the other way around? In service mode, the files are stored under OS-specific sysadmin account.

6

The full absolute file path was exactly the same between the original and reinstalled / reconnecting source data. The same physical disk was remounted to the same path that it was mounted to previously.

I’m not familiar with duplicati modes you are referring to. Is that the difference between the CLI and web interfaces or something else? I used the webUI in both the original and reinstall, on debian based systems using the .deb installation package and i did not intentionally change modes.

Since this scenario is not covered in the manual i would be really interested to hear from someone more familiar with the project than myself about what the current state of things is supposed to be, and whether there is any way to reconnect to the same backup.

The outcome i had could either be a bug or functionality that has never been implemented / considered.

7

Absolutely you can. You just need to configure the backup job the same as it was originally configured. Personally I save exports of job configurations and keep them in a safe place so I can import later if need be.

But this may be unnecessary as you stated above that you kept sqlite databases from the dead machine. You don’t “import” the databases in the Duplicati WebUI. Rather you would just shut down Duplicati and place the original sqlite files in the correct location. Default on Debian is /root/.config/Duplicati.

Duplicati-server.sqlite is pretty small and contains all the job configuration information, global settings, etc. The other sqlite files are job-specific databases and contain all the information about backed up files, blocks, hashes, etc.

After you place the files there, you can start Duplicati again and everything will just work.

If you have messed with the “back end” data at all since your system failure, you should undo that if possible. Otherwise it will be out of sync with the job-specific sqlite databases you recovered from your system at the time of failure. If you cannot undo any changes to the back end, then it may not be a huge deal, but Duplicati will notice something is “off” and ask you to run a repair.

8

To add a bit (in case you don’t have a full set of databases to move – that’s the easy way to reconnect), every “Add backup” job, whether “Configure a new backup” or “Import from a file”, will not know about a previous backup job database, so will make a new random pathname, editable in the Database screen:

image

If you don’t even have the old backup job database, you can use the Recreate button to recreate it, but ideally you have the old backup job database and just want to move it in. You can either rename it onto whatever new random path was invented (there won’t be anything there until initial backup), or edit the “Local database path” field after moving the old backup job’s database in. Editing activates the buttons, including “Save”, which (assuming all the moving and changing worked) will double-check. Click “Yes”:

image

If you think this should be better documented in the manual, I’d agree. I’m not even sure there’s a How-to, although if you’d like to double-check and refine this, anybody who’s willing/able to make one can make it.

What might be a good Feature request would be to ask if the UI could specifically lead one through things, however given quite a large backlog, some documentation approach might be faster to get some help out.

9

So if i’m understanding correctly, the truly fastest / easiest way to restore backups would be to copy both the Duplicati-server.sqlite as well as all of your .sqlite’s to the appropriate location (/root/.config/Duplicati) and then restart the web interface and all of your jobs would just magically re-appear.

If you don’t have Duplicati-server.sqlite or don’t want to completely wipe out other jobs on the system, but you can create the job using the exact same configuration that existed from before through import or memory, you can overwrite the new .sqlite with the old .sqlite or even change the path in the webui to point to the old .sqlite. One problem i have with this is that, if i have more than one of these database files, how am i goign to know which is which? I suppose it’s unencrypted and you could possibly find something useful inside the schema that tells you which backup it represents?

Perhaps there should be an option to import / export data from a duplicati-server.sqlite file so you could selectively extract the old configs from there and import them into the new isntall, or identify which file goes to which job?

I think this other option about recreating is really interesting because this seems like the least demanding on a user. Do i understand correctly that you must create a valid job configuration prior to doing this? Advanced parameters and block sizes must match or recreating will fail too? Why can’t i just point to my remote storage location and recreate everything?

I believe I may have been asked to run a repair on the database when i tried reconnecting. This sounds a lot like the same ‘recreate the database’ option. But for me, after re-creating the database, my old backups were getting deleted, so there is something risky about doing this, and maybe it would be especially risky if you haven’t restored your data to somewhere safe.

I would prioritize fixing the issue with duplicati deleting / losing data over any documentation or guided web UI’s, but if we can get any of those tasks done they all seem like they would have been pretty darn useful when i was going through this. Maybe someday soon i’ll get around to making some pull requests myself.

10

User mode is the default, so if you have not intentionally changed modes, you have been running Duplicati in user mode.

In user mode, the location /root/.config/Duplicati is not right.

All other questions you make are moot and answering them will confuse you even more until we have solved this puzzle. One we know which mode you are running Duplicati in, we can tell you where to place the files (stopping Duplicati before the move and starting it up after). Then you will see the jobs in the web ui and you will see which sqlite file matches which job.

11

I am sorry my previous message stopped short of explaining the difference of the modes. I was a bit busy, but now I can continue with that.

After installing Duplicati, there are two choices to start up Duplicati. If you launch Duplicati.GUI.TrayIcon.exe, either via icon or command line, you are running Duplicati in user mode, which is regarded as the default mode. If you want to switch into service mode, the command to launch Duplicati depends on the OS; there is one method for Windows (and only that one is explained in Different ways to make a Duplicati backup • Duplicati), one for Linux/Unix, and one for MacOS.

On Debian-based Linux, you have to run systemctl enable duplicati.service. But if you already had been using Duplicati in user mode, this new process, run as root, had to use different port for its web UI than the one already reserved for the user process (default is 8200). To connect to the service’s web UI, you need to modify the URL to use port 8300, or you may have to try out numers until you find the right one.

I have not used Duplicati in service mode myself, so I am not sure what happens if you never launch Duplicati in user mode, but immediately launch it in server mode; will it use the default port 8200 in that case? If that is the case, then it is entirely possible that a user may unwittingly launch Duplicati in service mode.

Now, equipped with this information, can you remember how exactly you launched Duplicati in the new installation? Whatever method you used, it must have been different from the earlier one.

EDIT: On Linux, the command duplicati is shortcut to Duplicati.GUI.TrayIcon.exe.