Release: 2.1.0.118 (Canary) 2025-05-12

Releases
1

2.1.0.118_canary_2025-05-12

This release is a canary release intended to be used for testing.

Major changes in this version

This version has a number of minor fixes and a major rewrite of the the “repair” command.
The logic for the “repair” command is that it should be possible to recover loss of .dblock files, if the data is still present locally.
This logic has been fixed in multiple ways and now also supports recovering data, even if no individual .dblock volumes can be fully recovered.
In this case, the repair will recreate as much data as possible in new .dblock files, and configure it so as many files as possible are available.

The purge-broken-files can be used after repair has failed to recover eveything, and will only purge the files that were lost.
The purge-broken-files command has also been updated to support using incorrect metadata, such that files are not purged if they are only missing metadata.

There are again numerous fixes to ngclient, including some faster browsing of restore contents, and better remote validation of folder contents.
The UI now supports a number of different languages.

Database version upgrade to v16

The local database is again upgraded with two index changes for correctness and performance.
The bundled Duplicati.CommandLine.DatabaseTool.exe / duplicati-database-tool can downgrade databases.
Since the change is only index addition, there is no data loss on downgrades.

Detailed list of changes:

  • Support CACHEDIR.TAG as a default exclude file marker
  • Improved list-broken-files to more accurately show contents, thanks @gpatel-fr
  • Added new faster API for listing restores (only used in ngclient)
  • Added new API for testing to check for existing files (only used in ngclient)
  • Updated translations, thanks to all the translators
  • Fixed pCloud OAuth url in CLI
  • Improved logic for combining Regex filters, thanks @Jojo-1000
  • Improved error parsing for box.com backend
  • Simplified log closing to avoid cases where the log file is kept open
  • Prevent database actions when delete is invoked with no versions to delete
  • Updated throttle library to force more smooth output
  • Tracking task metadata (start/stop time) for better log scoping
  • Fixed AuthID not being reported as a password property
  • Removed CloudFiles backend
  • Fixed issue with throttle not working correctly on some backends
  • Fixed an issue with rclone giving errors after each operation
  • Fixed repair command to support repairs with partial data available
  • Updated local DB schema to v16 (two new indexes)
  • Fixed scheduling order when starting a backup
  • Fixed case where warnings were emitted if the local data contains duplicates during repair, thanks @warwickmm and @Jojo-1000
  • Updated iDrivee2 to use HttpClient
  • Updated TahoeLAFS to use HttpClient
  • Removed long-standing FIXMEGlobal class
  • Fixed issue with server-util timing out after 15 min, if using the --wait option

Ngclient changes:

  • Fixed “Export as commandline”
  • Prevent GUI commandline from sending empty arguments
  • Fixed some options were missing from the commandline view
  • Added some confirmation dialogs
  • Added indicator to show if backup is encrypted
  • Improved notification state not always showing errors
  • Fixed issue with multiple request fired on restore browsing
  • Updated restore to use new faster API, if available
  • Fixed issue with percent-encoded paths from configuration import
  • Fixed issues with Test button not detecting SSL certificates or SSH key changes in all places
  • Updated the Test button to check for existing files if the API is available
  • Stop restore attempts early on known fatal errors (passphrase missing, empty folder, etc)
  • Fixed an issue with advanced option lists not showing the correct labels
  • Added a restore progress page
  • Support multiple root folders on restore
  • Test destination has a spinner while active
  • Added spinners for long-running tasks from the start page
  • Added TahoeLAFS UI
  • Fixed the Live logs area
  • Added multiple languages to the UI, thanks to all the translators
  • Updated login page and loading indicator
3 Likes
2

Started on the upgrades from .117 - had a nice candidate to start with, RPI-4 that showed an error during last night’s backup. Upgraded, re-ran backup and same error:

 Warnings 1 
2025-05-13 07:56:55 +02 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-ExtraUnknownFile]: Extra unknown file: duplicati-ba541b5e130db409b8f9a1278584a6b66.dblock.zip.aes
 Errors 1 
2025-05-13 07:56:55 +02 - [Error-Duplicati.Library.Main.Controller-FailedOperation]: The operation Backup has failed with error: Found 1 remote files that are not recorded in local storage. This can be caused by having two backups sharing a destination folder which is not supported. It can also be caused by restoring an old database. If you are certain that only one backup uses the folder and you have the most updated version of the database, you can use repair to delete the unknown files.
RemoteListVerificationException: Found 1 remote files that are not recorded in local storage. This can be caused by having two backups sharing a destination folder which is not supported. It can also be caused by restoring an old database. If you are certain that only one backup uses the folder and you have the most updated version of the database, you can use repair to delete the unknown files.

Ran a repair, it logged:

  "Messages": [
    "2025-05-13 07:58:29 +02 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Repair has started",
    "2025-05-13 07:58:37 +02 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  ()",
    "2025-05-13 07:58:37 +02 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed:  (2.898 KiB)",
    "2025-05-13 07:58:38 +02 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Started: duplicati-b4fe242a683dc450a89e25c1d450aebad.dblock.zip.aes (14.000 MiB)",
    "2025-05-13 07:58:38 +02 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Completed: duplicati-b4fe242a683dc450a89e25c1d450aebad.dblock.zip.aes (14.000 MiB)"
  ],

Re-ran the backup, it failed again, this time:

 Warnings 1 
2025-05-13 07:57:57 +02 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-ExtraUnknownFile]: Extra unknown file: duplicati-b4fe242a683dc450a89e25c1d450aebad.dblock.zip.aes
 Errors 1 
2025-05-13 07:57:57 +02 - [Error-Duplicati.Library.Main.Controller-FailedOperation]: The operation Backup has failed with error: Found 1 remote files that are not recorded in local storage. This can be caused by having two backups sharing a destination folder which is not supported. It can also be caused by restoring an old database. If you are certain that only one backup uses the folder and you have the most updated version of the database, you can use repair to delete the unknown files.
RemoteListVerificationException: Found 1 remote files that are not recorded in local storage. This can be caused by having two backups sharing a destination folder which is not supported. It can also be caused by restoring an old database. If you are certain that only one backup uses the folder and you have the most updated version of the database, you can use repair to delete the unknown files.

Re-ran the repair:

    "2025-05-13 07:57:14 +02 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Repair has started",
    "2025-05-13 07:57:22 +02 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  ()",
    "2025-05-13 07:57:22 +02 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed:  (2.899 KiB)",
    "2025-05-13 07:57:22 +02 - [Information-Duplicati.Library.Main.Operation.FilelistProcessor-IgnoreRemoteDeletedFile]: Ignoring remote file listed as Deleted: duplicati-b4fe242a683dc450a89e25c1d450aebad.dblock.zip.aes",
    "2025-05-13 07:57:24 +02 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Started: duplicati-ba541b5e130db409b8f9a1278584a6b66.dblock.zip.aes (6.000 MiB)",
    "2025-05-13 07:57:24 +02 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Completed: duplicati-ba541b5e130db409b8f9a1278584a6b66.dblock.zip.aes (6.000 MiB)"

Re-ran the backup and it completed the backup without issues.

Going to test a Windows machine next to see if the upgrade causes settings to reset again.

3

Looks a bit strange, just wanted to make sure it was not a copy-error, but:

  • 1st error: duplicati-ba541b5e130db409b8f9a1278584a6b66.dblock.zip.aes
  • 1st delete: duplicati-b4fe242a683dc450a89e25c1d450aebad.dblock.zip.aes
  • 2nd error: duplicati-b4fe242a683dc450a89e25c1d450aebad.dblock.zip.aes
  • 2nd delete: duplicati-ba541b5e130db409b8f9a1278584a6b66.dblock.zip.aes

So it looks like it complains about file 1, deletes file 2, complains about file 2 (which was just deleted), then deletes file 1.

Is this accurate?

4

Ah shoot, I copy pasted them in the wrong order - you can see it by the timestamps. Sorry about that.

5

Arh yes, I see, then it makes more sense.

I still don’t understand how it can report 1 extra file on the first run, and then 1 (different) extra file again on the second run?

Both remote “extra” files should be there the whole time, unless something else is touching the storage. Is it possible that there was some non-Duplicati activity on the folder?

6

Just noticed the ngclient “last run” dates/times for each backup has mixed accuracy for the “actual” option (mostly wrong), but the “relative” option displays “a month ago”. The ngax client has the correct dates/times listed (last Sunday for all).

7

It’s an SMB share (caching disabled) on another Windows server dedicated for storing backups so I don’t think so, I definitely don’t back up that folder, I have Defender exclusions for each Duplicati folder to prevent it interfering with the files there and to help with performance.

8

Thanks for reporting.
It is tracked in this issue.

The code that does this check looks like this:

So if it is unlikely that this is something external popping in files, the second .dblock file must have been removed from the database as part of the repair somehow.

1 Like
9

So I was forced to replace my RPi-4 that died and switched it to a spare NUC and running what was on the RPi-4, Zabbix, as a Proxmox LXC using Debian. The rebuild has gone well and now I have also gotten Duplicati transferred across. It then reminded of a service start warning I have had for a while:

May 16 09:17:40 ZIFF duplicati-server[8877]: SQLite warning (284): automatic index on B(BlocksetID)
May 16 09:17:42 ZIFF duplicati-server[8877]: SQLite warning (284): automatic index on G(BlocksetID)

Any idea what this is? I suppose it’s referring to the main database not one of the job databases.

10

This is for the job databases.
It is a warning that SQLite is creating an index because the query needs it.

I have a fix for that one specifically, if you see others, let me know.

1 Like
11

I can no longer delete old backups wonders if that was on Canary, gives 2.1.0.118 failure where --no-encryption=True is sent from Commandline when backup encryption is on.

Looking at release more, Status bar progress is present but is off-screen past second job. Scrolling to top of window is possible, but ngax fixed header at top, regardless of job view.

This will become more of an issue if job view ever has anything dynamic like file progress, however I’m not certain the ngax view was truly correct, given the parallelism of a backup.

Source tree does now seem to show what’s selected, but in too much detail, so very slow. Rather than expand a tree as needed per UI click, my backup issues about 50 all at once, getting mostly successes, about 11 500 response codes, and eventually showing huge list.

This is my small carefully curated production backup. I wonder how awful a big one looks?

Is “Revert to NGAX client” going to be the standard terminology? To most people, NGAX is meaningless. Since ngax offers “Use new UI”, should ngclient show offer as “Use old UI”?

I’m not sure if Edit of Advanced options is supposed to show them, but it generally doesn’t. Results seem to be a little intermittent though, and I’m not sure why. Commandline as well.

I had ngclient come up in Commandline once without a command up top or in bottom, but similar issue would sometimes happen in ngax too. It came up in help not backup, IIRC.

EDIT:

In case it’s not clear, these are ngclient commments unless said otherwise.

12

ngclient Commandline seems to have an Advanced options count or vertical space cutoff.
My regular backup was missing some options (I have quite a few) I know should be there.

Testing with small test backup with no advanced options, after 15 adds, the rest vanished.
I suspect they’re there but invisible because if I delete an option higher up, they go visible.

Commandline displays current Advanced options in a tiny field, bad for reading or editing.
There is no description available of what option does. It does reveal option name though.
Maybe this part doesn’t know descriptions? It seems to think everything is custom option.

image

Adding a new option reverses these problems. The missing part here is the option name:

image

EDIT 1:

It does display a few pre-existing options in a nicer way, but in general they are “Custom”.

image
image573×246 15.3 KB

13

Thanks for reporting these. I have registered individual issues for them.

14

I filed two ngclient issues directly.

Destination log lacks ngax “Load older data” button #227

Destination log text is mostly not copyable, copies caret-down. #228

Another could use some discussion about Duplicati results reporting.

Missing contents in log page #82

“Additionally (maybe same issue) the warnings are not showing” which got fixed, but what about error count, and what about fatal, and what icon and color should be used for what?

image
image487×390 15.2 KB

The top backup has ParsedResult of Fatal, maybe explaining icon, text, and no “1 error”.

On ngclient, it looks like this:

image
image363×182 8.38 KB

which made me first wonder about no info on errors, then wonder about icon and color.

It wasn’t clear how to get a ParsedResult of Error instead of Fatal. Anyone got a case?

Improve result reporting #4978 was old discussion. I’m pretty sure ngclient lacks these, however a developer should look to see exactly how much of the ngax function got lost.

Possible code for for of this which I found before the ngax PR (which also cites its files):

https://github.com/duplicati/duplicati/blob/master/Duplicati/Server/webroot/ngax/templates/backup-result/entryline.html

https://github.com/duplicati/ngclient/blob/main/src/app/backup/log/general-log/general-log.component.html

15

Restore version picker gives wrong version number
Restore selection tree doesn’t highlight selections
restore shows files as if they are folders
restore restores excess files whose names begin with selected file
restore does not restore file timestamp
are now filed on 2.1.0.119 for these issues. Maybe the notes below will be helpful.

ngclient restore

  • shows wrong version numbers (is it fileset ID?), but does restore chosen one
  • doesn’t highlight restore selects, but does seem to notice what was selected
  • shows files like they are folders, but unsurprisingly only folders really expand
  • restores excess files, maybe due to considering them folders, so appending *
  • does not set file time at default setting unlike ngax, and maybe at any setting

ngax view:

image
image721×228 12.4 KB

ngclient view:

image
image543×246 11.9 KB

When I was first poking at this, I had a simpler setup, and caught some restore traffic:

ngax did:

{time: "2025-05-26T20:02:20-04:00", restore_path: "C:\backup restore", overwrite: true,…}
overwrite
: 
true
passphrase
: 
""
paths
: 
["@C:\backup source\short.txt"]
permissions
: 
false
restore_path
: 
"C:\\backup restore"
time
: 
"2025-05-26T20:02:20-04:00"

ngclient did:

{paths: ["C:\backup source\short.txt*"], passphrase: null, time: "2025-05-27T00:02:20Z",…}
overwrite
: 
true
passphrase
: 
null
paths
: 
["C:\backup source\short.txt*"]
permissions
: 
false
restore_path
: 
"C:\\backup restore"
skip_metadata
: 
true
time
: 
"2025-05-27T00:02:20Z"

EDIT 1:

Seeing it ask for short.txt* inspired my short.txt.zip. Ask for short.txt, get both.
I’m hazy on CLI restore syntax, but wouldn’t a true folder request have \ near path end?

EDIT 2:

Testing that,

ngax asks for
“paths”:[“[C:\\backup source\\FOLDER_A\\.*]”]

ngclient asks for
“paths”:[“C:\backup source\FOLDER_A\*”]

I’m not sure which is right, but both seem to work (except for file timestamps in ngclient).

16

Maybe we can discuss ngclient Alerts versus Notifications, now that they’re operational.

I’m not a UI designer, but Google gave me its automatic AI Overview. I asked Perplexity.

Both make me wonder if Duplicati is using these backwards (in addition to excessively).

An alert is something urgent, needing action, often in response to something user asked.
The ngax popups sort of fit this model, were a little annoying, and were extremely visible.

These now seem to show up as Notifications, available by an icon with a bell (for alarm?).

image

The i inside a circle seems like it’s commonly used for more information, so may fit badly.
To me, seeing it conveys a sense of “FYI”, meaning no urgency or handling now required.

So why are they called “Alerts” and the (alarm?) bell is called “Notifications”. Backwards?

Regarding “excessively” comment, the so-called “Alerts” are largely new noise to dismiss.
Many of them may be routine http activity. And there’s no way to dismiss them all at once.

EDIT 1:

For an example of what is probably a nuisance alert, I edit a job. Comes up fine, but says:

image
image359×255 9.93 KB

which is probably just an expired token, handled fully transparently except for this “Alert”.
Admittedly some of the messages might be things to worry about, but most seem benign.

EDIT 2:

is probably what persists the ngax red (error) and yellow (warning) popups, and it uses “notification” for both. I’m looking at words more from an external interface point of view.

A historical choice was to use “Notifications” for the popups. If this is still the preference, maybe a word that isn’t “Alerts” could be found for whatever the nuisance information is.

Or maybe filtering it would give it a better signal-to-noise ration, but what’s it for anyway?

17

I don’t think this distinction is very clear, and has little value for the user IMO.
The backup failed, if it was a expected or unexpected error is mostly a developer thing.

I made a commit that fixes this

Thanks! :folded_hands:

I think all issues in this topic are now registered as issues (or fixed).

We are currently working with a designer to overhaul the notifications.

Yes, this is a flaw in how the API was designed back when jQuery was the way everybody worked. The API sends error messages for things that are not really errors (e.g., sends 404 when asked if a file exists). This trips up error handling that is common today. We are suppressing these error messages when found, and have started on the API v2 that does not have these quirks.

The intention is to give the user some feedback if the server responded unexpectedly with an error (like a timeout for instance). This explains why the UI stops responding, and hopefully suggests what to do. But I agree that in the current form, they are mostly noise because they are triggered even when the error condition is handled.

Not sure what the best way for this is, but the idea was/is that users may not have the UI open. If they open it, it should be very visible that something is not running as expected. At least for home users this is important, more tech-savvy people have set up some kind of monitoring so they are alerted.