Release: 2.0.6.1 (beta) 2021-05-03

kenkendk · May 3, 2021, 6:46am

2.0.6.1-2.0.6.1_beta_2021-05-03

First beta build in over a year!

We have been hard at work adding new features and fixing issues. This beta release is a roll-up of all the fixes since the last beta build and is expected to cause very few issues when upgrading. This release is based on 2.0.6.0_experimental.

A big thanks to all Duplicati contributors, supporters, fans and users for keeping the project evolving and improving.

New in this version

Added Tardigrade backend
Added Tencent Cloud Object Storage (COS)
More Docker image versions
Improved long-path support on Windows

We changed some larger things that you should be aware of when upgrading.

The SSH backend has been updated

This update increases the security by prefering stronger encryption algorithms.
However, this can cause failing backups as the host key is reported as “changed”.
If this happens, edit the backup and update the host key.

Database version update from v10 to v11!

Downgrade from this version requires manually adjusting the version number in the database. The additions can be re-applied if the database is upgraded again later.

Other changes that needs attention

Duplicati now requires Mono 5.10 or greater
S3 and Sharepoint option names on commandline now uses - for consistency (previously used _)

tacioandrade · May 3, 2021, 6:02pm

I heard that the new versions of Duplicati have improvements in the re-creation of the database and things like that. I would like to know if I update an older beta version (from last year), if it would be recommended to start the backup job from scratch or can I leave it as it is and continue with the existing retentions and have a better performance than the old ones just by changing the version of the agent used to restore.

marcosvrs · May 4, 2021, 12:14am

I lost all my backup configs as it created a new sqlite db when updated.
How to recover the lost config? The old sqlite db is still in the /data folder.

ts678 · May 4, 2021, 12:29am

Welcome to the forum @marcosvrs

What OS is this, and is this a Docker? If so, which? I don’t use any, but Duplicati’s requires this setup:

https://hub.docker.com/r/duplicati/duplicati

Preserving configuration

All configuration is stored in /data inside the container, so you can mount a volume at that path to preserve the configuration:

$ docker run --name=duplicati -v duplicati-data:/data duplicati/duplicati

This allows you to delete and recreate the container without losing your configuration:

$ docker rm duplicati
$ docker run --name=duplicati -v duplicati-data:/data duplicati/duplicati

ts678 · May 4, 2021, 12:44am

I’m not sure about re-creation improvements, but the database should be a little more reliable in 2.0.6.1.

I’m not sure restore performance will improve through code change, however if that’s a major goal, then fresh start might restore faster for awhile for files that have changed, because history collection is easier.

marcosvrs · May 4, 2021, 12:49am

Hi @ts678,

It’s running under docker in Ubuntu 18.04 bionic.
Yes, I’m running the docker with locally hosted volume:

duplicati:
    image: duplicati/duplicati
    volumes:
        - ./.docker/duplicati/data:/data
        - /home/marcosvrs:/home
    networks:
        - local
    restart: unless-stopped

And after the update, my data folder looks like that:

.docker/duplicati/:
total 4.0K
drwxr-xr-x 4 root root 4.0K May  3 23:00 data

.docker/duplicati/data:
total 8.0K
drwxr-xr-x 3 root root 4.0K May  4 00:00 Duplicati
drwxr-xr-x 3 root root 4.0K May  3 23:00 .mono

.docker/duplicati/data/Duplicati:
total 2.8M
-rw------- 1 root root 445K May  2 23:00 'backup ID3 20210503110000.sqlite'
-rw------- 1 root root 103K May  3 00:00 'backup ID4 20210504120000.sqlite'
drwxr-xr-x 2 root root 4.0K Apr 30 13:48  control_dir_v2
-rw------- 1 root root  34K May  4 00:00  Duplicati-server.sqlite
-rw------- 1 root root  66K May  3 02:00  ID2.sqlite
-rw------- 1 root root 776K May  3 23:01  ID3.sqlite
-rw------- 1 root root 1.2M May  3 01:00  ID1.sqlite
-rw------- 1 root root 122K May  4 00:00  ID4.sqlite

.docker/duplicati/data/Duplicati/control_dir_v2:
total 4.0K
-rw-r--r-- 1 root root 3 May  3 22:07 lock_v2

.docker/duplicati/data/.mono:
total 4.0K
drwxr-xr-x 3 root root 4.0K May  3 23:00 certs

.docker/duplicati/data/.mono/certs:
total 4.0K
drwxr-xr-x 2 root root 4.0K May  3 23:00 Trust

.docker/duplicati/data/.mono/certs/Trust:
total 0

ts678 · May 4, 2021, 1:32am

You probably need someone who uses Docker. Let me see if @drwtsn32 is available to help with this.

One thing you can maybe confirm is my guess that the ID# are the ten-random-letter backup filenames.
Maybe times to the second would help. It gets to those databases via Duplicati-server.sqlite, but if there genuinely is nothing there, how did it find the backup databases? Also, what’s special about 00:00 time?

I suppose you could grep --text \<a-backup-sqlite-name> on the DB to see if you can find it there.

kenkendk · May 4, 2021, 2:04pm

You mention that you “lost all my backup configs”.

This indicates that the file Duplicati-server.sqlite has been deleted/cleared for some reason.
However, the backup IDx ...sqlite files indicates that you have been able to run the updated Duplicati at some point. The files are created because Duplicati needs to update the database layout, and it makes a copy prior to altering the database.

The size is 34K which indicates that it is not empty, and should contain your backup configurations.

My best guess is that you somehow have 2 versions of Duplicati running, and are connected to the other instance, which has a clean database. This would explain that you have the automatic backups running at night, which would update the database and create the copies you see.

ts678 · May 4, 2021, 2:46pm

This is why I’m asking @marcosvrs what IDx means. I suspect they’re job database backups with random ID characters. I forget what a Duplicati-server.sqlite backup is called, but it shouldn’t happen because there was no version upgrade to cause backup (whereas there was for the job databases).

So we know no file names, and no precise file times. ls -l --full-time can show precise times.

Size of backup ID3 is less than non-backup ID3, and its date is older. Might be OK, but May 2 date is
before 2.0.6.1, so what was it on? The system looks to be in in ahead-of-UTC timezone, as it shows
May 4 dates on a (to me) May 3 post. Or maybe the clock is just set wrong. What time zone is it in?

Simple grep for one of the ID# strings in it would help. Heavy approach is install and run sqlitebrowser.

Web interface empty task list when not on local machine is a weird waiting-for-originator problem that sounds like server systeminfo data didn’t arrive, so it looks like jobs aren’t there on some browsers.
There are other screens broken as well, so it should be possible for @marcosvrs to try that testing.

marcosvrs · May 4, 2021, 4:16pm

Thank you all for the effort!
After recreating the docker container, I can see my configs again. Really weird!

One thing you can maybe confirm is my guess that the ID# are the ten-random-letter backup filenames.

Yes, I just didn’t want to share too much

I suppose you could grep --text \<a-backup-sqlite-name> on the DB to see if you can find it there.

Unfortunately, the command returns empty. Don’t know what I did wrong.

However, the backup IDx ...sqlite files indicates that you have been able to run the updated Duplicati at some point.

The weird part is that now I have 4 backup database files:

.docker/duplicati/data/Duplicati:
total 4.3M
-rw------- 1 root root  66K May  3 02:00 'backup MFAMIQGHEG 20210504020000.sqlite'
-rw------- 1 root root 445K May  2 23:00 'backup MILNJBGBOD 20210503110000.sqlite'
-rw------- 1 root root 1.2M May  3 01:00 'backup OQVEHHMFDP 20210504010000.sqlite'
-rw------- 1 root root 103K May  3 00:00 'backup XXXFDBUMNR 20210504120000.sqlite'
drwxr-xr-x 2 root root 4.0K Apr 30 13:48  control_dir_v2
-rw------- 1 root root  34K May  4 02:00  Duplicati-server.sqlite
-rw------- 1 root root  82K May  4 02:00  MFAMIQGHEG.sqlite
-rw------- 1 root root 776K May  3 23:01  MILNJBGBOD.sqlite
-rw------- 1 root root 1.5M May  4 01:00  OQVEHHMFDP.sqlite
-rw------- 1 root root 122K May  4 00:00  XXXFDBUMNR.sqlite

Another info that might be important for discovering what happened is that I run the https://containrrr.dev/watchtower/ container to auto-update my other containers. Which happened to be the one updating Duplicati.

I can see all my backups stored safely and up to date.
From my side, I consider this case closed. But I’m happy to provide more infos if you want to continue debugging this case to see what went wrong.

ts678 · May 4, 2021, 4:48pm

ls -lcrt --full-time on the folder might help, and I think will pick up the time when a file is renamed. I might try moving this discussion to its own topic because I don’t want to make the release notice unworkably long…

marcosvrs · May 4, 2021, 5:38pm

$ ls -lcrt --full-time .docker/duplicati/data/Duplicati/
total 4384
drwxr-xr-x 2 root root    4096 2021-04-30 13:48:05.182514713 +0000  control_dir_v2
-rw------- 1 root root  455680 2021-05-03 23:00:00.440846366 +0000 'backup MILNJBGBOD 20210503110000.sqlite'
-rw------- 1 root root  794624 2021-05-03 23:01:09.282901708 +0000  MILNJBGBOD.sqlite
-rw------- 1 root root  105472 2021-05-04 00:00:00.078587382 +0000 'backup XXXFDBUMNR 20210504120000.sqlite'
-rw------- 1 root root  124928 2021-05-04 00:00:17.407143006 +0000  XXXFDBUMNR.sqlite
-rw------- 1 root root 1234944 2021-05-04 01:00:00.468584838 +0000 'backup OQVEHHMFDP 20210504010000.sqlite'
-rw------- 1 root root 1547264 2021-05-04 01:00:20.461187557 +0000  OQVEHHMFDP.sqlite
-rw------- 1 root root   67584 2021-05-04 02:00:00.082129122 +0000 'backup MFAMIQGHEG 20210504020000.sqlite'
-rw------- 1 root root   83968 2021-05-04 02:00:19.258666557 +0000  MFAMIQGHEG.sqlite
-rw------- 1 root root   34816 2021-05-04 02:00:19.286667342 +0000  Duplicati-server.sqlite

aschrage · May 6, 2021, 9:09am

Hello all, first of all thanks for the new beta version. Duplicati is my number one backup solution

Now to my “problem” with the update from my current version 2.0.5.1.
I did the update in the GUI and also activated it. Now in the Duplicati update folder I can see the new version 2.0.6.1. Also in the GUI it states the new version and in the protocol of the backup job it says Version": “2.0.6.1 (2.0.6.1_beta_2021-05-03)
Now to my confusion.
I have installed Duplicati as Service on my Windows machine. When the service starts it points to this EXE:
C:\Program Files\Duplicati 2\Duplicati.WindowsService.exe” “–server-datafolder=C:\ProgramData\Duplicati\Data”
This EXE still says that it is version 2.0.5.1
I thought when I activate the new version the update gets copied to the program files folder, but there are still all the the files from the previous version.

Can someone help me with my confusion?

kenkendk · May 6, 2021, 9:33am

That is how it is supposed to work. The updater works such that when you call the .exe it will start by looking for a newer version. If one such is found, it will pass its input to the newer version. This causes some confusion as you can see multiple .exe processes listed in the Process Explorer.

aschrage · May 6, 2021, 9:51am

Thanks for the quick reply. Now I understand.
I checked the currently running Duplicati service and it indeed points to the EXE in the update directory.

Is it advised to do a clean install with the new version sometime in the future?
What happens when I do command line operations outside the GUI, should I point this to the update directory or will it use the same functionality and redirect it to the newer version in case I use the command line from the program files folder?

kenkendk · May 6, 2021, 10:36am

No. That should not have any effect.

You should always point to the base version (i.e. the one originally installed). There are a few items that are read from that folder and not from the update folder.

Rob_Canada · May 7, 2021, 4:49pm

It’s always great to see a new release come out! However, I’m having a small difficulty that I’m sure would be an easy fix if I had a better understanding of it.

I’m running the Synology build on a DS720+ and it has been working great. My backup device is a WD Passport that doesn’t support many protocols. It does, however, work with SFTP, which is what I’ve been using.

Since updating, the backup has been failing. I edited the backup config to see if anything had changed and realized that when I try to test the connection, I get “Failed to connect: The method or operation is not implemented.”

I read the mention of the SSH backend changing, so I deleted the key that I had stored with the backup config, just in case that was an error. Ultimately, no change in behaviour.

Does anybody have any ideas?

If I can get it working then I’ll have to downgrade, in which case I’ll need to figure out how to change the database version manually.

ts678 · May 7, 2021, 5:20pm

Release: 2.0.6.1 (beta) 2021-05-03 - SSH: Validation of server fingerprint failed has some.
I’d request that that be the place for followups, so as not to make this thread too cluttered.

Rob_Canada · May 7, 2021, 5:58pm

Thank you. I have given that stuff a read and will create a new thread.

rpitera · May 12, 2021, 1:49pm

Like @marcosvrs I too lost my config/backup settings, but I’m on Win10. When I got the update notification, I downloaded and attempted to install, but after the progress bar was finished, it just took me back to the first notification. I attempted again, and got the same notification to update so I dismissed it and closed the WebUI thinking I would go back to it later.

Ironically enough I got a Windows update notification shortly after, so I applied it and waited for the update to finish. When I went to look at Duplicati, my configs and backup job were gone. It looks like my SQLite db was backed up (indicating to me that the install had completed - right?) but I’m at a loss on how to proceed.

I really like Duplicati and I understand beta woes (I’m involved in other OSS projects like HomeAssistant and Joomla) but I’d really hate having to recreate all that work, not to mention having to delete and recreate the backups I’ve stored on OneDrive.

Any ideas on how to either get the db restored or roll back would be greatly appreciated; I’d put a lot of time into filters and email notifications and had everything working great until I tried to upgrade.