Preview of security and bugfix release Canary 105

1

Hello

The binaries for Debian/Windows/Mac are here:

The source code zip has been generated by Github, I have no idea of what is inside. Probably not the PR I have used (see here: synthetic proposed changes for canary 105; by gpatel-fr · Pull Request #18 · gpatel-fr/testdupl · GitHub)

Note that this is NOT a standard release, it’s not intended to be used long term.
I will delete these binaries in 5 days.
It is only published to allow experienced users of this forum to provide
help by giving feedback on the platforms/setups that I could not test (see below)
before wider exposition through the Canary channel.

Please also read the usage advice below.

Included changes (read more about them on Github):

library updates

PR#4897 Mega
PR#4901 SSH NET
PR#4857-4877 Newtonsoft

fixes

PR#4779 Jottacloud disable V2-authID
PR#4802 Tray icon mac (+add licence bouncy castle, PR 4903)
PR#4902 Debian bookworm install
PR#4815 exclusions
PR#4879 faster hashing
PR#4905 Impossible Cloud

Experimental changes

PR#4895 rewrite of directory listings
PR#4898 database rebuild
PR#4907 custom database options (Sqlite pragmas)

Usage

to avoid any risk, it’s best to install on a new computer (VM) and use test data
if you want to take the risk to test this software on your own data, that’s your call.
if you use it on someone else’s data, I prefer not to have anything to do with you.

in case of tests on an existing install, remember the following points:

the database structure is compatible with current canary (104) NOT with current beta.
This means that when if install this binary preview over a beta, it will be necessary to
recover with the canary 104

This is not a release to keep used for some time. Automatic upgrade will not work.
For reasons (bad reasons), the displayed version in the UI is 2.07.
The simplest way to install may be to stop Duplicati, save the current content
of the Duplicati app directory, unzip the zip file in it, restart Duplicati.
When the tests are finished, do the reverse.
The installers should work also but it may not be the simplest way (under
Windows it may be asked to restart the computer if the service mode has
been set)

Personal tests

Note that I am not dumping a random compilation
on people wanting to help test this, I have done some basic checks.
However, I can’t test everything and that’s where you can help.

Sftp: Ubuntu 20.04 LTS, backup 80Mb, restore 2 files
S3 (Minio): Ubuntu 20.04 LTS, backup 7Gb, restore 1 file
Webdav: Win serv 2016, backup 70Mb, restore 1 file
OneDrive: Ubuntu 20.04 LTS, backup 50Mb, restore 1 file
Mega: backup 15 Mb, restore 1 file
Storj: Win10, backup 100 Mb, restore 1 file

Experimental features

I have included 3 experimental features to take the opportunity to
have some feedback on it. If I don’t have any I will not commit them.

These experimental features are only activated by setting environment variables, so if you
don’t set them, Duplicati behaviour will not be affected.

  • custom sqlite pragma:
    env var EXPERIMENTAL_CUSTOMSQLITEOPTIONS_DUPLICATI
    set to list of SQLite pragmas (don’t include the word ‘pragma’) separated by ‘;’
    check it is active: use ‘list’ function in the Web UI and set log to verbose, see messages
    impact: hopefully better performance across the board
    risks: if you set an invalid syntax it will just generate an error message; if you try to set a variable that will conflict with Duplicati behaviour, you risk bad problems. Stick to EXPERIMENTAL_CUSTOMSQLITEOPTIONS_DUPLICATI=cache_size=-200000 and you should be safe (remember that the unit is 1 KB, so -200000 means 200 MB, a value that may be trivial for your computer or definitely too much if you use Duplicati on a nanocomputer such as a Raspberry Pi)

  • list database:
    env var EXPERIMENTAL_DBLIST_DUPLICATI
    set to 1
    if you have a database with >1M blocks, the time taken to display a directory
    with >500 entries in the Web UI Restore files browser should be markedly better.
    risks: very low if you don’t do backups, since the impacted code will only be in
    the web user interface.

  • recreate database:
    env var EXPERIMENTAL_DBRECREATE_DUPLICATI
    set to 1
    you can recreate a new database without touching your current database from the command line, setting or not the environment variable and compare the time taken.
    See details here: EXPERIMENTAL rebuild database optimization by gpatel-fr · Pull Request #4890 · duplicati/duplicati · GitHub
    risks: obviously this is the most dangerous option.

If you start Duplicati from your own session, set the env variable for your user
account and start Duplicati.
If you installed Duplicati as a service:
For a Windows service, configure the service entry in the registry
create a MultiSZ variable named ‘Environment’, then add lines of the form VARNAME=VARVALUE
For a Linux service, configure the environment variables in SystemD with
[Service]
Environment=VARNAME=VARVALUE
In all cases you need to restart the service for the environment variables to
be used.
When you recover your normal installation, the environment variables can stay set,
they are not impacting current Canary (104)

2

The Windows .msi installer seems to now need an uninstall first. 2.0.6.104 didn’t have this need.
Maybe this is a result of a different build process for the preview? Without uninstall, only .exe are:

 Directory of C:\Program Files\Duplicati 2

03/18/2023  02:32 PM            13,312 mozroots.exe
03/18/2023  02:32 PM            37,888 SharpAESCrypt.exe
               2 File(s)         51,200 bytes

To confirm installer results, I uninstalled, and installed 2.0.5.1, 2.0.6.3, 2.0.6.104 OK then preview.
Duplicati was down at the time and was not started during any of the installations. I just ran a dir.

3

Yes you are correct I never tested to install over an existing setup, always either uninstall / install, or just copy over files extracted from the zip file. It does not seem to work indeed. Surprising but I’m not going to fix that for the time being, the goal of this preview is not the replace the current mechanism, there are more urgent things to do - such as cut a release…

4

Another workaround is Apps & Features → Duplicati 2 → Modify → Repair. It puts in the missing files.
Safer in general (and apparently needed here) is Duplicati 2 → Uninstall before installing this preview.

Theory:

Windows Installer sees the default 2.0.0.7 version number on preview files, and refuses to downgrade.
Testing the official builds, 2.0.6.103 installed over 2.0.6.104 had the same issue. The files that actually made it in don’t use the regular numbering plan, so might have escaped some “downgrade” blocker…

Testing:

Installs and uninstalls of Duplicati. Once for Windows service, plus a sanity test to see something work. Did a small restore with Advanced option no-local-blocks. Without that, it just takes blocks from source.
About 300 MB each over FTP and FTP (Alternative). Moved larger B2 and OneDrive backups onto this.
Those backups are set up to look for issues, so run a compare, a test, a database recreate, and more.

I’m starting test with what’s handy on my Windows install, but sometime it might be worth asking others, because some of the fixes require specific OS or destination. So far so good here after getting installed.

5

I tested EXPERIMENTAL_DBLIST_DUPLICATI separately with a find command. It ran 10x slower.
This backup lists roughly 12000 files and folders if I ask for everything (e.g. giving it a “*” wildcard).

I don’t have any giant folders to expand in the Restore GUI. What I had seemed roughly the same.

I put all three enhancements in larger backup described above, and got about 10% faster recreate compared with EXPERIMENTAL_DBRECREATE_DUPLICATI not enabling recreate enhancement.

6

Thanks for the feedback

Ugh. I need to investigate why this happens. I am not sure I tested a list with all files (that’s not something very useful, I did tests with selecting ‘big’ directories, a few hundred of files, among a lot of files). Maybe it would be useful to raise cache as the query would get a lot more data in this case.

how large exactly ? what’s the output of select count(*) from blocks ? If it’s below 500000, I would not expect the performance of Duplicati to be too bad anyway, so improvements would be minimal. And the impact is greater when there are bad/missing index files because the problematic queries are run again and again.

7

I don’t have a giant backup handy. This one was only about 84000 blocks, but I do have one from same source with lots more versions that’s about 263000, which is about a quarter way to recommended max.

Let me get some more timings with what I have. Ultimately (but most work), special test data might help.

8

The OneDrive backup is a recent fresh start for writing the look-for-issues script. I’ll just lower blocksize, maybe by factor of 20 (so use 5 KB blocksize instead of 100 KB). That will probably slow the recreate…

9

For such a small backup I’m actually surprised you got even 10% better time. I suspect it was mostly coming from the general effect of a greater cache size.

that’s what I did for these tests to turn a 10 GB backup into a 100 GB simulated one.

10

Possibly. One of the testing perils of too many potential combinations of things…

5 KB blocksize backup solved that by setting optimization just before repair step.
First test result shows a 16% reduction in time, both running with a larger cache.
Actual times were 172 versus 205 seconds, so still not in the horribly slow range.
Recreate is from a local hard drive OneDrive rclone. Blocks are about 1,280,000.

Backup seemed a lot slower, and it seemed like it might have gotten CPU limited.
Hard drive activity seemed rather light. Two profiling logs were being hugely busy,
however they help more to see slow queries (not a general-purpose code profiler).

I could extract the repair from its current after-backup context (also saves run time).
Hiding a random dindex file is easy enough, however what I’d prefer would be that
Duplicati gets better at noticing things like missing dindex files as part of its checks.

Repair can recreate dindex if DB has data. I can force it with a Remotevolume edit.
In fairness, it’s good once, but I think (haven’t tested just now) that after a recreate
wrestles a database from dblock file reads, it just stays happy with bad destination.
Present but corrupted dindex files are a harder problem, and all of this is wishlist…

11

It’s actually logical given the problem you have found in the find (aka list) command.
That’s a thing a backup has to do: listing all known files and Duplicati is doing just that with this function, that was better in my tests and is worse in your case.

With db rebuilding, the reduction you are seeing is actually not surprising me. The time taken don’t come only from one query, it has first a fixed time, downloading and handling all the list and index files, and this is not changed by this query, and then when handling the missing index files by this query that is slow and getting exponentially slower when the database size (block number) grows (a 2 TB data size to backup means 2M blocks at default size), and the fact that it’s done for each missing / bad index file. With a healthy backend, it’s only done one time, but with a damaged backend, it can go bad.

About lists, the list environment variable is enabling 2 different functions, the plain list (bad results in your case, that’s the one used in backups) and folder contents list (used in the Web UI, for which you did not find any difference). Maybe I could enable them separately.

12

This is done in ListFilesHandler, which I think is also used by GET /api/v1/backup/2/files
during GUI restore to list new folder expansion. Why would backup do this? Its listing is in source.

After backup, if it wanted to see what files were in new Fileset, it could look in FilesetEntry for that
FilesetID, and take FileID set to File view. Maybe it already does. I haven’t gone looking in code…

In addition, work was crawling, not just at the end, and I see even less why backup constantly lists.
One theory is that I should have reduced dblock size, as there are now a whole lot of blocks in one. Possibly this makes the space calculation (test-if-full) get kind of slow, and there are lots of those…

I see that (and I also see that the naming of these variables has been changing), but it still looks like ListFilesHandler is needed, and I haven’t found the path there from backup. Can you give a hint?

Regardless, I’ll return to obtaining some timings. This is getting beyond what a batch file does easily.

To anyone else who is watching, feel free to test it at any level you like, even if it’s a very simple one. Getting a working Canary out that fixes what it claims to would be good. I’d rather not need a re-spin.

13

Sorry, there is no path from backup to this SelectFiles function that I changed in this experimental way. I assumed that but I missed that it’s really never called in a normal backup, only in test mode.
So if you see a ‘crawling’, I think it’s due to the higher block number. The only thing that could possibly impact the Duplicati core backup way in the changes is the Json library updating, and it seems very unlikely that it could have this effect. Can you confirm that with this higher block count, the previous Canary is faster and NOT ‘crawling’ ?

14

Well, I just tested again the list (find) command with my biggest test bed. It terminated in 9’19’’ with EXPERIMENTAL_DB_LIST_DUPLICATI. I ran it immediately after with = 0 (inactive) and it is still running since 30’. I have killed it since I need to go to sleep, but how it is that you got a so different result ? It’s very strange.

15

It will have to be overnight, due to time required. Also, I’m currently measuring various recreate flavors.
I’m not having any luck forcing it to pass 3 by deleting random dindex. Do you understand the passes?
I’m looking for the best way to break the backup heavily, as we seemingly sometimes get on the forum.

I’ll test with list some more, but first here are some times from different recreate options and damage:

Missing 0 dindex
Processing indexlist volume 109 of 109
Recreate completed, verifying the database consistency
seconds dbrecr  dblist  cache
441     0       1       0
453     0       0       0
387     1       0       0
197     1       0       1

Missing 1 dindex
Processing indexlist volume 108 of 108
Probing 1 candidate blocklist volumes: duplicati-b884bd38e22af42a5b0512c58b0be190a.dblock.zip.aes
Probing 1 candidate blocklist volumes
Backend event: Get - Started: duplicati-b884bd38e22af42a5b0512c58b0be190a.dblock.zip.aes (50.00 MB)
  Downloading file (50.00 MB) ...
Backend event: Get - Completed: duplicati-b884bd38e22af42a5b0512c58b0be190a.dblock.zip.aes (50.00 MB)
Pass 2 of 3, processing blocklist volume 1 of 1
Recreate completed, verifying the database consistency
seconds dbrecr  dblist  cache
551     0       1       0
548     0       0       0
447     1       0       0
203     1       0       1

Missing 2 dindex
Processing indexlist volume 107 of 107
Probing 2 candidate blocklist volumes: duplicati-b6447ce32ccff44f6889e94bb36b2b541.dblock.zip.aes, duplicati-b884bd38e22af42a5b0512c58b0be190a.dblock.zip.aes
Probing 2 candidate blocklist volumes
Backend event: Get - Started: duplicati-b6447ce32ccff44f6889e94bb36b2b541.dblock.zip.aes (50.00 MB)
  Downloading file (50.00 MB) ...
Backend event: Get - Completed: duplicati-b6447ce32ccff44f6889e94bb36b2b541.dblock.zip.aes (50.00 MB)
Backend event: Get - Started: duplicati-b884bd38e22af42a5b0512c58b0be190a.dblock.zip.aes (50.00 MB)
  Downloading file (50.00 MB) ...
Pass 2 of 3, processing blocklist volume 1 of 2
Backend event: Get - Completed: duplicati-b884bd38e22af42a5b0512c58b0be190a.dblock.zip.aes (50.00 MB)
Pass 2 of 3, processing blocklist volume 2 of 2
Recreate completed, verifying the database consistency
seconds dbrecr  dblist  cache
664     0       1       0
694     0       0       0
520     1       0       0
232     1       0       1
16

I don’t get the meaning of the array you posted (with seconds / dbrecr / dblist / cache - I guess that dbrecr and dblist are operations, but what’s cache ?)

About the passes, it’s done in DoRun routine in Duplicati/Library/Main/Operation/RecreateDatabaseHandler.cs

line 188 downloading the file lists is step 1
line 329 reading the index files is step 2
line 439 reading the missing blocks is step 3
this step 3 is indeed done in 3 passes

By deleting index files, you are hitting step 3 of DB recreation, but not necessarily using the 3 passes of the step 3. Yes it’s a bit confusing :slight_smile:

The third pass of the step 3 is supposed where all not yet read blocks are downloaded until the remaining indexes have been recreated. My guess is that it’s when downloaded blocks at second pass are revealing more index data. I have not yet seen this pass being hit.

17

Coming back to this performance discrepancy, I can confirm it. I had done most of my performance tests under Linux, and not noticed the performance gap between Linux and Windows. Under Windows, the list (find) performance is a lot better than under my Linux setup. And indeed setting the list environment variable under Windows gives much worse results as you reported.

However, when using --list-folder-contents (that’s the function used in the Web UI), I don’t see the same as you, even under Windows. The performance for the same query goes down from 40s to 7s when going from Linux to Windows, but under Windows with the new query I get 5s, and adding cache (-200000) it goes down to 4s, it’s not an enormous win but still noticeable.

18

Below is an expanded description of that.

dbrecr is EXPERIMENTAL_DBRECREATE_DUPLICATI value
dblist is EXPERIMENTAL_DBLIST_DUPLICATI value
cache 1 is EXPERIMENTAL_CUSTOMSQLITEOPTIONS_DUPLICATI=cache_size=-200000
cache 0 is EXPERIMENTAL_CUSTOMSQLITEOPTIONS_DUPLICATI= (to be sure it’s clear)

Reading result rows going down is actual tests order, so possibly first one had less OS cache.
Each set of 4 tests is a batch file run. Damage (removed dindex) was raised with time on runs.
There is a sample of the ending of recreate log shown to see the reaction to missing dindexes.

The code I linked to is what does pass analysis, but SQL was too tough for me to understand.
I’m pretty sure I can force it to read all dblocks by hiding a dblock. Would that help you much?

Duplicati.CommandLine.exe help list-folder-contents
  --list-folder-contents (Boolean): Show folder contents
    When searching for files, all matching files are returned. Use this option
    to return only the entries found in the folder specified as filter.
    * default value: false

I’d say help list should show it too, yet it does not. Strange. Anyway, my CLI test didn’t set it.
GUI test of expanding a folder passed in something that sounds similar. The full text of a get is:

GET /api/v1/backup/2/files/C%3A%5CUsers%5CMaintenance%5C?prefix-only=false&folder-contents=true&time=2023-03-20T12%3A50%3A02-04%3A00&filter=%40C%3A%5CUsers%5CMaintenance%5C HTTP/1.1

My overnight 2.0.6.104 backup is even slower than the roughly 6 hour one on preview 2.0.6.105.
I’m watching what I think are future dblock files growing, and total growth is about 92 KB/second.
Basically, no clear evidence that preview 2.0.6.105 is worse. I don’t recall if initial backup was on enhanced cache or not. If it was, that might explain why it was faster than current very slow pace.

19

If I understand you correctly, the cache rise had more effect than the new query.

not sure about that, I’d say that Duplicati would register a missing dblock instead.
You’ll notice at line 601 of LocalRecreateDatabase.cs that after each pass, there is a check that all blocks (hashes) are accounted for. If the index file is read successfully, all hashes are known, so all blocksetentries, blocklisthashes are known. So pass 2 can be successful even if a dblock file is missing and pass 3 is never done. The data inside is not known but there is no need to read all the (other) data files, it would be pretty pointless.

20

I hid the dblock for a previously hidden dindex, basically making all its blocks not easy to find.
Duplicati read all the dblock files trying. This was headed towards a three hour run, so I killed.

26 minutes with dbrecreate and large cache
94 minutes with dbrecreate only. This is still better than the killed run with just dblist turned on.
Based on the tests so far, it looks like that option doesn’t affect recreate, so I might stop using.

I see I need to log stderr to the file too. The log file only sees it start verification. Terminal gets:

ErrorID: DatabaseIsBrokenConsiderPurge
Recreated database has missing blocks and 3 broken filelists. Consider using “list-broken-files” and “purge-broken-files” to purge broken data from the remote store and the database.

So there’s the report. Not sure what to do next. If you want something specific, please describe.