Web GUI lost all backup configurations on Linux, even though db is present

Hi
I’m not familiar with duplicati under linux but your trouble sounds like a problem of user context under duplicati runs: obviously duplicati use the enviroment of the user which launch the process. If you go under “about” -> “sytem info” and you should see the field “username”, next you can move the directory according to the username field or edit the startup script and use the command line option --server-datafolder. Good news: according to your post you haven’t lost your backup jobs :slight_smile:

UserName : root

My Duplicati folder with the sqlite files is owned by my user.
Since Duplicati is run by root, I think it should be able to access the files nevertheless.
I could try moving or changing owner or permissions, but to be owned by root, then?

On a side note, I seem to understand that a lot of bugs have been fixed the last few months. Would changing to experimental channel theoretically be of any advantage, or disadvantage?

Duplicati search his files in /root/.config/Duplicati but your file are in $HOME/.config/Duplicati. It’s true: duplicati run as root and it don’t have permission related problems but the files are in the wrong place. So you can move the directory .config/Duplicati in root’s home or specify the path when the data are actually stored.

I don’t know the roadmap of developers but as my experience there are many important bug fix and improvements from last beta and the bug fix are in the latest experimental but seem that a new beta should be released soon.

1 Like

Duplicati search his files in /root/.config/Duplicati but your file are in $HOME/.config/Duplicati. It’s true: duplicati run as root and it don’t have permission related problems but the files are in the wrong place. So you can move the directory .config/Duplicati in root’s home or specify the path when the data are actually stored.

Thanks @xblitz, when I copied the files over to /root/.config/Duplicati the web GUI displays it.

However, I get errors about not finding the files to back up. This is because the initial backup was based on my user, so it was configured to back up $HOME/bin etc. Since /root does not have a bin/ folder, an error is issued.

I do want Duplicati to work based on my own user’s homedir. So I guess I’ll have to look into how to set up the service to use --server-datafolder option.

How come this changed between reboots? First I run with files in my $HOME for weeks, and after the next reboot it changes to /root ?

seem that a new beta should be released soon.

Can you fill me in on how to spot such indications? I have tried looking around, but I have seen nothing indicating an imminent new beta release.

I can’t know how is configured your system but $HOME is a enviroment variable the --server-data-folder don’t affect this parameter the parameter depend to the user which launch the process (root in your case now). IMHO the simplest way is to configure the backup to point to the directory you want backup (use absolute path).

Unfortunately this is only a rumor, anyway:

NOTE FOR DEVELOPERS: I hope in a new beta version as soon as possible so i can start to advise this software to relatives and friends :slight_smile: :slight_smile: :slight_smile:

Here was my post on the link you gave. Not sure if you want to talk others into being pre-Beta testers, however feel free to run it yourself. One never knows what testing may show up, but so far so good…

This Beta probably won’t be going Stable but at least it will probably be much better than current Beta.

Unfortunately many of my friends aren’t involved in IT so I must propose them reliable solutions: I use the canary channel but the update process is frustrating (I talk about bugs related to the update process when duplicati run as service) and sometimes, for me, useless (I don’t use onedrive, google team, amazon s3)… of course in my PC’s I can do what I want but I can’t in PC of other people :blush: and if somethings go wrong the software stop to work… stop to backup… (bad thing). Finally I can’t advise them the current beta version because it means give up to improvements (splecially related to the rebuild process of databases).

Duplicati search his files in /root/.config/Duplicati but your file are in $HOME/.config/Duplicati. It’s true: duplicati run as root and it don’t have permission related problems but the files are in the wrong place. So you can move the directory .config/Duplicati in root’s home or specify the path when the data are actually stored.

OK, that makes sense. I try to do as suggested, specify backup paths absolute and not relative to $HOME. Regardless of what the backup db path is ($HOME/.config/Duplicati/86757982797878868475.sqlite or /root/.config/Duplicati/86757982797878868475.sqlite) I still get the same error:

Finished!

            
Backup started at 1/10/2020 8:39:26 AM

ErrorID: DatabaseRepairInProgress
The database was attempted repaired, but the repair did not complete. This database may be incomplete and the backup process cannot continue. You may delete the local database and attempt to repair it again.
Return code: 100

Via the Web GUI I have tried Database → Repair, Database → Recreate, to no avail. I have also tried deleting the sqlite file and then doing repair as the error message suggests, but it does not help.

Any suggestions for how to proceed?

If nothing else can be done I can try creating a new backup config from scratch, but having this experience and not understanding why it happens does not increase my confidence in the product. But I really want to have confidence in it.

Some more detailed logging below.

Log level profiling (without sensitive details):

Backup started at 1/10/2020 8:50:46 AM

The option log-level is deprecated: Use the log-file-log-level and console-log-level options instead

The operation Backup has started

Starting - Running Backup

Starting - ExecuteScalarInt64: INSERT INTO "Operation" ("Description", "Timestamp") VALUES ("Backup", 1578642646); SELECT last_insert_rowid();

ExecuteScalarInt64: INSERT INTO "Operation" ("Description", "Timestamp") VALUES ("Backup", 1578642646); SELECT last_insert_rowid(); took 0:00:00:00.009

Starting - ExecuteReader: SELECT "Key", "Value" FROM "Configuration"

ExecuteReader: SELECT "Key", "Value" FROM "Configuration" took 0:00:00:00.000

Starting - ExecuteReader: SELECT "Key", "Value" FROM "Configuration"

ExecuteReader: SELECT "Key", "Value" FROM "Configuration" took 0:00:00:00.000

Starting - ExecuteScalarInt64: SELECT COUNT(*) FROM "Block" WHERE "Size" > 102400

ExecuteScalarInt64: SELECT COUNT(*) FROM "Block" WHERE "Size" > 102400 took 0:00:00:00.000

Starting - ExecuteReader: SELECT "Key", "Value" FROM "Configuration"

ExecuteReader: SELECT "Key", "Value" FROM "Configuration" took 0:00:00:00.000

Starting - ExecuteReader: SELECT "Key", "Value" FROM "Configuration"

ExecuteReader: SELECT "Key", "Value" FROM "Configuration" took 0:00:00:00.000

Running Backup took 0:00:00:00.014

HTTP Response: 201 - Created: {"username":"<cut>","ip":"<cut>","backupsetname":"<cut>","result":"Failed","errors":"Duplicati Backup report for <cut>\n\nFailed: The database was attempted repaired, but the repair did not complete. This database may be incomplete and the backup process cannot continue. You may delete the local database and attempt to repair it again.\nDetails: Duplicati.Library.Interface.UserInformationException: The database was attempted repaired, but the repair did not complete. This database may be incomplete and the backup process cannot continue. You may delete the local database and attempt to repair it again.\n at Duplicati.Library.Main.Operation.BackupHandler+<RunAsync>d__19.MoveNext () [0x0012b] in <e2da9713f0974e76879d9f9aa7ce0e36>:0 \n--- End of stack trace from previous location where exception was thrown ---\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x0000c] in <8f2c484307284b51944a1a13a14c0266>:0 \n at CoCoL.ChannelExtensions.WaitForTaskOrThrow (System.Threading.Tasks.Task task) [0x00050] in <6973ce2780de4b28aaa2c5ffc59993b1>:0 \n at Duplicati.Library.Main.Operation.BackupHandler.Run (System.String[] sources, Duplicati.Library.Utility.IFilter filter) [0x00008] in <e2da9713f0974e76879d9f9aa7ce0e36>:0 \n at Duplicati.Library.Main.Controller+<>c__DisplayClass13_0.<Backup>b__0 (Duplicati.Library.Main.BackupResults result) [0x00035] in <e2da9713f0974e76879d9f9aa7ce0e36>:0 \n at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String[]& paths, Duplicati.Library.Utility.IFilter& filter, System.Action`1[T] method) [0x0011d] in <e2da9713f0974e76879d9f9aa7ce0e36>:0 \n\nLog data:\n2020-01-10 08:50:46 +01 - [Warning-Duplicati.Library.Main.Controller-DeprecatedOption]: The option log-level is deprecated: Use the log-file-log-level and console-log-level options instead","backupset_id":14932,"updated_at":"2020-01-10 08:50:47","created_at":"2020-01-10 08:50:47","id":4519088}

Whole SMTP communication: Connected to <cut>
<cut>

Stacktrace from E-mail report:

at Duplicati.Library.Main.Operation.BackupHandler+d__19.MoveNext () [0x0012b] in :0
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x0000c] in <8f2c484307284b51944a1a13a14c0266>:0
at CoCoL.ChannelExtensions.WaitForTaskOrThrow (System.Threading.Tasks.Task task) [0x00050] in <6973ce2780de4b28aaa2c5ffc59993b1>:0
at Duplicati.Library.Main.Operation.BackupHandler.Run (System.String sources, Duplicati.Library.Utility.IFilter filter) [0x00008] in :0
at Duplicati.Library.Main.Controller+<>c__DisplayClass13_0.b__0 (Duplicati.Library.Main.BackupResults result) [0x00035] in :0
at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String& paths, Duplicati.Library.Utility.IFilter& filter, System.Action`1[T] method) [0x0011d] in :0

Picture is still murky, but let me throw in some questions and comments.

How do you enter that? It’s not in the tree of checkboxes, and Add path field won’t take it when I try.

–server-datafolder is for keeping the server SQLite files. It is not related to getting the user’s homedir. Windows users who want TrayIcon for their Windows service set --no-hosted-server true. The service cannot interact with user’s desktop. I’m less familiar with Linux, but issue and answer may be similar.

If it was running as you for weeks, then perhaps you ran /usr/bin/duplicati which starts the Tray Icon. Reboot lets systemd get there before you, and often start as root, depending on the systemd config, which would let you set another user (some distros do it – I forget exact how to do), if that helps any.

Another difference visible in systemd status post is that it starts /usr/lib/duplicati/Duplicati.Server.exe, whereas /usr/bin/duplicati runs Duplicati.GUI.TrayIcon.exe, which is Server plus Tray Icon combined.

The above TrayIcon documentation also shows how it runs a search sequence for other ports, if the usual port 8200 is busy. For example, Server could get 8200 first, then Tray Icon gets 8300 and user gets confused by finding nothing at 8200. It “should” at least give a clue of offering password setting, however after doing that the user wonders why they’re in but all the backups are gone. Not good UX.

What does it do? Any error messages on the Recreate? You posted backup errors, but what about this?

There was (and maybe still is) some confusion about what database was being used. It should be what the Database screen shows.as Local database path field, and whatever user Duplicati is running as needs access. After you delete and Repair, does it come back or does any file change in the directory?

“The database was attempted repaired, but the repair did not complete.” looks like it means backup run found a database that’s marked as not having completed a repair, meaning it’s seeing a database then.

Going step by step, I’d prefer seeing Recreate run before Backup gets in, but Backup can start by itself when Duplicati starts if it’s a scheduled backup that missed a schedule. You may want to unschedule it until a database can be obtained. So let’s first see if Recreate will work. You can watch action at About → Show log → Live → Verbose if you like. It should download dlist, dindex, and maybe dblock files to recreate the database. Here’s a Recreate live log on 2.0.5.0 Experimental. Newest activity is at the top:

Jan 10, 2020 2:02 PM: The operation Repair has completed
Jan 10, 2020 2:02 PM: Recreate completed, and consistency checks completed, marking database as complete
Jan 10, 2020 2:02 PM: Recreate completed, verifying the database consistency
Jan 10, 2020 2:02 PM: Pass 1 of 3, processing blocklist volume 1 of 1
Jan 10, 2020 2:02 PM: Backend event: Get - Completed: duplicati-b2d92328d5c4643948c4db8a3dcba8127.dblock.zip (55.99 KB)
Jan 10, 2020 2:02 PM: Backend event: Get - Started: duplicati-b2d92328d5c4643948c4db8a3dcba8127.dblock.zip (55.99 KB)
Jan 10, 2020 2:02 PM: Processing required 1 blocklist volumes
Jan 10, 2020 2:02 PM: Processing required 1 blocklist volumes: duplicati-b2d92328d5c4643948c4db8a3dcba8127.dblock.zip
Jan 10, 2020 2:02 PM: Processing indexlist volume 1 of 1
Jan 10, 2020 2:02 PM: Backend event: Get - Completed: duplicati-i4ef1d70cfd574521ad93b37c792c9808.dindex.zip (7.59 KB)
Jan 10, 2020 2:02 PM: Backend event: Get - Started: duplicati-i4ef1d70cfd574521ad93b37c792c9808.dindex.zip (7.59 KB)
Jan 10, 2020 2:02 PM: Filelists restored, downloading 1 index files
Jan 10, 2020 2:02 PM: Processing filelist volume 5 of 5
Jan 10, 2020 2:02 PM: Backend event: Get - Completed: duplicati-20200108T035738Z.dlist.zip (2.20 KB)
Jan 10, 2020 2:02 PM: Backend event: Get - Started: duplicati-20200108T035738Z.dlist.zip (2.20 KB)
Jan 10, 2020 2:02 PM: Processing filelist volume 4 of 5
Jan 10, 2020 2:02 PM: Backend event: Get - Completed: duplicati-20200108T035734Z.dlist.zip (2.20 KB)
Jan 10, 2020 2:02 PM: Backend event: Get - Started: duplicati-20200108T035734Z.dlist.zip (2.20 KB)
Jan 10, 2020 2:02 PM: Processing filelist volume 3 of 5
Jan 10, 2020 2:02 PM: Backend event: Get - Completed: duplicati-20200108T035731Z.dlist.zip (2.20 KB)
Jan 10, 2020 2:02 PM: Backend event: Get - Started: duplicati-20200108T035731Z.dlist.zip (2.20 KB)
Jan 10, 2020 2:02 PM: Processing filelist volume 2 of 5
Jan 10, 2020 2:02 PM: Backend event: Get - Completed: duplicati-20200108T035727Z.dlist.zip (2.21 KB)
Jan 10, 2020 2:02 PM: Backend event: Get - Started: duplicati-20200108T035727Z.dlist.zip (2.21 KB)
Jan 10, 2020 2:02 PM: Processing filelist volume 1 of 5
Jan 10, 2020 2:02 PM: Backend event: Get - Completed: duplicati-20200108T035722Z.dlist.zip (2.20 KB)
Jan 10, 2020 2:02 PM: Backend event: Get - Started: duplicati-20200108T035722Z.dlist.zip (2.20 KB)
Jan 10, 2020 2:02 PM: Rebuild database started, downloading 5 filelists
Jan 10, 2020 2:02 PM: Backend event: List - Completed: (7 bytes)
Jan 10, 2020 2:02 PM: Backend event: List - Started: ()
Jan 10, 2020 2:02 PM: The operation Repair has started

Beta Duplicati versions are not as informative in the log, in addition to lacking about a year of bug fixes.

One way to sanity-test your backup without worrying about what database it’s using is to see how far a direct restore can go on a sample of files restored to some other folder (avoid overwriting original files).

It will just make a partial temporary database in a temporary area, but that will prove such a feat works.

This will also simulate a disaster recovery. Now, you can copy configuration from another tab, but if the drive is lost then you should probably have an Export of the configuration saved somewhere that’s safe.

It shows as “User data” in the web GUI. See screenshot below.
duplicati-sourcedata-home

Something feels odd to me here. It does not feel right presenting root’s homedir as “User data” just because the service runs as root. I assume this is a scenario seen in Linux context before, though I am not too familiar with it.

In the future I will avoid “User data” and instead use “Computer” only.

Yeah, that probably explains it, thank you!
I don’t remember exactly how the install process was, but on Linux it should probably recommend an immediate reboot if the Duplicati service is installed. I don’t think it did.

I hope I can get the tray icon to become visible. I don’t know how to edit the command-line used by the service, and I am not sure I want to do that either, because it will make upgrades, reinstalls and maintenance more complicated. Instead, I am hoping that a newer version will fix it. The mentioned Duplicati Tray Icon When Running as Linux Service indicates some bugfix in the area.

Recreate did bring back an .sqlite file as far as I can tell, the error still showed. But no need to dwelve more on that, see below.

I had several backups configured, and I am in the process of getting them up and running or starting fresh, mostly done now I think. All backups now have database path in /root/.config/Duplicati. Some smaller ones work fine, but the largest still has issues.

It is no longer the issue seen earlier, I can only repeat that with a backup that has files selected, but nothing remote yet. That one is going from error “Repair did not complete” on backup, to error “No files were found at the remote location” upon recreate. No worries, I’ll replace the backup configuration with a new one. For now all scheduling is disabled.

The issue with the largest backup at the moment is as follows:

Checking errors, related to #1400. Unexpected result count: 0, expected 1, hash: Ze3lQbMsZNbYzpMJpjYQBi9tXVuqotle9MZFcgdcmnk=, size: 102400, blocksetid: 365279, ix: 18, fullhash: Iii82/Tzh5Uob8WW46qGnOlV8ozs6M3Ir0+Tv0jR77g=, fullsize: 2036840
2020-01-17 08:23:08 +01 - [Error-Duplicati.Library.Main.Database.LocalBackupDatabase-FoundIssue1400Error]: Found block with ID 2554155 and hash Ze3lQbMsZNbYzpMJpjYQBi9tXVuqotle9MZFcgdcmnk= and size 25376
2020-01-17 08:23:08 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: /root/.config/Duplicati/86757982797878868475.sqlite-journal

When I moved the database file from $HOME to /root via Database → “Move existing database” it automatically started repair or recreate (not sure which), and this ran for a couple of days (the backup source is 47 GB). Then I tried a backup, and got the above error. I have alternative backup for the moment for those files, so I can replace that one as well.

I’ll try out newer versions (experimental) when I get time.
Thanks for all the useful information! I did get the configurations in the web GUI back after all :+1:

Linux is potentially problematic (compared to Windows at least) due to all of its possible variations.
There are different package formats with different scripting ability, changing different start systems.

Some people may also dislike reboots. I know I do. So ideally one recommends a gentler solution, however this varies by the particular OS, much as one might like to do it in portable Duplicati code.

The reboot (or service stop/start, or whatever) needs are also likely different for an upgrade install. There are also potentially multiple upgrade paths – through Duplicati, or through OS package tool.

Beyond this, anything built by third-party packagers is beyond direct control, but we could model…

Have you thought about this enough to suggest specific portable ways to do this? I know I haven’t.

If you refer to the Sep 20, 2019 post on Canary made here, fix would seem to be only shown below:

v2.0.4.30-2.0.4.30_canary_2019-09-20

Fixed a crash when connecting to a password protected server instance from the trayicon, thanks @kenkendk

There was just a tray icon discussion here with other references. It’s a different program that you run, which then connects to the server, and interacts with you. On Windows, I don’t think a service can do user interaction. I’m not sure about Linux. Maybe it depends on how you set up X Windows security?

Perhaps security also gets in the way of user messages you requested to guide user at initial install? The whole installation area is kind of rough at the moment, and that includes Windows service install.

CheckingErrorsForIssue1400 and FoundIssue1400Error test case, analysis, and proposal #3868

Fix for issue1400 #3872

v2.0.4.22-2.0.4.22_canary_2019-06-30

Fixed sporadic issue with backups of files being written, thanks @BlueBlock

SQLite journal file is a common case hit the fixed “files being written” bug because it’s written a lot…

Hello!

Updated yesterday to Duplicati - 2.0.5.112_canary_2021-01-20 on Manjaro, and today I see all my backup configs disappeared. It’s like a completely clean slate, it even asked me for password protection, even though I had already a password set up.

I don’t have a /root/.config/duplicati or ~/.config/duplicati folder, so I don’t know where to look for my configs. Can you help me?

I am a bit frustrated with this to be honest. Can you also tell me if this is expected behavior or if it happens often?

I was really fond of Duplicati up until now, I even had it configured for 2 non-techy friends, but this may be an instant deal-breaker for me.

For any further info that may help you help me, please ask.

Welcome to the forum @Chuhtra

How did you do the original install, and how did you update? The AUR packagers had Duplicati run as user duplicati not user root. This throws off a lot of people. Updating Duplicati through Duplicati seems like it will not upset whatever you had before, but I’m not positive the same is true for update using an AUR package.

https://aur.archlinux.org/packages/duplicati-latest/ updated yesterday, so the timing is a little bit suspicious.

Is that typed right? It should be Duplicati not duplicati. Also look in ~duplicati/.config/Duplicati

Installing Duplicati on Linux (Arch / Manjaro) talks about how to edit your systemd unit file if you prefer root.

Another thing that sometimes can happen is you get two copies of Duplicati up. You can look at browser’s URL. If it’s port :8200, that’s the first Duplicati. If it’s :8300 it’s a later second-run copy, and could be empty.

EDIT:

In any Duplicati GUI you can also look at About → System info at UserName to see what user that’s using.

Hi @ts678, thank you for the quick response.

Yes this is the package I am using, which I updated through pamac, so not through Duplicati but through the package.

Yes the file /root/.config/Duplicati does exist and has some files in it, so I am either stupid or yesterday was very tired cause while I thought about capitalization I didn’t even think that ls -l doesn’t show hidden files. Sorry for that.

Yes this thread rings a bell. So since right now the duplicati.service file has duplicati as user and group and the GUI location you mentioned agrees as well, my read is that the files I just mentioned are the files I had deemed lost. Am I right?

Interesting. I checked and I don’t have the copy running.

Now supposing that these are indeed my files, I would like some tips (or suggestions to other threads) regarding 1) how I will return them in my running duplicati, connect them to the location of actual backups etc. (how I will be up and running again more or less) and 2) moving forward what practices I should follow with regard to updating Duplicati and backing up its configuration, so I won’t be put in the same spot again?

Thanks a ton for your rational take on my situation, you most probably got me out of a lot of trouble.

You mean it says UserName : duplicati? That makes it look in ~duplicati/.config/Duplicati and see nothing.
You also won’t have permission to your files unless you did something like add them to duplicati group.

If you found somewhat recently dated files in root’s folder, that’s probably the backup you were on before.
I’m not expert in systemd and know nothing of pamac. I hope you don’t have to fix systemd each update.
Unless you know you’re set up to run duplicati:duplicati, maybe kill those two lines and restart duplicati…

OK, I simply changed duplicati.service username and group, restarted duplicati service with systemctl command, and now it found my files. I connected succesfully and did a backup. Only hickup is that the “There is an available update” message was up, while I was on the “new” version, so I just dismissed it.

Do you know if it’s possible to change username and group (let’s say make it duplicati:duplicati) without having to reconfigure the jobs? I guess if I did that, future updates wouldn’t worry me.

You want the same files backed up? Then no job changes, but you need to adjust Linux to allow access.
I think what AUR had in mind was using secondary groups, e.g. see How to Add User to Group in Linux.
This all seems a little non-obvious, and maybe you could post a comment (or look for one) on their plan.
The original motivation, I think, is that running as root is too powerful, especially if network-connected…

Another option that might work depending on your backup needs is to have Duplicati run as you. You can move the randomly-named job files to whatever new user you like, but Duplicati-server.sqlite has full path information to the jobs, so you’d probably want to export and import each job into a new job configuration.

I think I get it, but a bit vaguely. Could you maybe point me to some duplicati documentation that would be helpful or relevant with this? I don’t want to completely overtake this thread or take advantage of your time.

Exporting a backup job configuration to get a job configuration from previous user into an external file.
After moving a job (and maybe tests), don’t run it from the old location, and consider deleting old one.
EDIT: because you never want multiple jobs going to same destination, as they’ll confuse each other.
Creating a new backup job to get it into new user, which will begin empty (just like yours was empty).
Database management to find which database goes with old job, and to get a copy into relocated job.
Duplicati components to show how multiple Duplicati can coexist on different ports (different browser
seems to help them keep from tripping over each other, which I suspect is a bug but haven’t reported.
Or if you prefer, you can just run one at a time, then Quit in tray icon right-click menu. This is all more
complicated as a Linux service though (e.g. no Tray Icon). You can stop service and start in a shell as
duplicati (/usr/bin/duplicati to be exact) to get one running as whatever user you’re then running as.

1 Like

Thanks a lot! I will check them out.