Release: 2.0.9.108 (Canary) 2024-10-03

kenkendk · October 3, 2024, 5:25pm

2.0.9.108_canary_2024-10-03

This release is a canary release intended to be used for testing in preparation of a later stable release.

Unlike regular canary builds, this one has a major change in the build system, so it now runs on .NET8.

For that reason, the updater in previous canary builds does not detect this update yet, but this can be activated at a later time.

The builds are self-contained so Mono or .NET installations are not required to install.

Important changes from last Beta

Updated to .NET8 with OS specific builds
Using Kestrel as the API/UI server
Mandatory password and new authentication scheme for server
Settings database version updated to v8

Please see list of known issues related to .NET8/Kestrel upgrade:

New tool to manage a running server

Due to incompatibility with duplicati_client a new tool is included, named Duplicati.CommandLine.ServerUtil.exe/duplicati-server-util.

Upgrade from `v2.0.9.105`

If you are upgrading from 2.0.9.105 please see the release notes from 2.0.9.106 for how to prepare the database.
Upgrades from other versions do not need special steps.

New experimental UI

This version features a new more modern UI based on Angular18
Not all features are implemented just yet, so the new UI is not yet default.
The curious ones can try it by visiting http://localhost:8200/theme.html (or similar).

Detailed list of changes:

Added export feature to server-util
Default to using .NET built-in Zip code for performance improvements
Added /health healthcheck API to server + server-util
Removed unuspported use of Thread.CurrentCulture
Removed unsupported use of Environment.OSVersion
Removed support for piped download as it was slow and caused hangs
Fixed an issue with multi-instance detection that caused crashes
Added a new experimental UI
Added support for enabling remote control

ts678 · October 3, 2024, 6:01pm

Release: 2.0.9.107 (Canary) 2024-09-11 raised a lot of issues, yet had no developer response. Possibly some comments may yet emerge, either here, or where issues were originally raised.

Presumably many of the issues that are 2.0.9.x issues still exist, but some others might be just problems that are not new. If possible to tell the difference, I’d encourage only 2.0.9.x problems reported on the Canary release announcement because all support here gets too cumbersome.

Past topic got very messy due to multiple long discussions on different things, at the same time, causing at least one subthread in the topic to break off into a new topic, so lose its history there.

If development has a preferred way on how to handle things for 2.0.9.108, please suggest here.

drwtsn32 · October 4, 2024, 2:40am

Trying this on my Debian trixie laptop. Upgrading from 2.0.9.105, so I did the database decryption step.

When I run a backup, I now get an SSL error. My backup job is targeting my Synology NAS using WebDAV+SSL with a self-signed cert. Backup job is configured with the --accept-specified-ssl-hash option and the fingerprint is still correct (matches the cert on my NAS). It has worked this way for a long time, just not with 108.

I also tried --accept-any-ssl-certificate and it still failed.

Any ideas?

Error:

System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure: RemoteCertificateNameMismatch, RemoteCertificateChainErrors
   at System.Net.Security.SslStream.SendAuthResetSignal(ReadOnlySpan`1 alert, ExceptionDispatchInfo exception)
   at System.Net.Security.SslStream.CompleteHandshake(SslAuthenticationOptions sslAuthenticationOptions)
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken)
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Duplicati.Library.Main.BackendManager.List()
   at Duplicati.Library.Main.Operation.FilelistProcessor.RemoteListAnalysis(BackendManager backend, Options options, LocalDatabase database, IBackendWriter log, IEnumerable`1 protectedFiles)
   at Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList(BackendManager backend, Options options, LocalDatabase database, IBackendWriter log, IEnumerable`1 protectedFiles, Boolean logErrors)
   at Duplicati.Library.Main.Operation.BackupHandler.PreBackupVerify(String backendurl, Options options, BackupResults result)
   at Duplicati.Library.Main.Operation.BackupHandler.RunAsync(String[] sources, IFilter filter, CancellationToken token)
   at CoCoL.ChannelExtensions.WaitForTaskOrThrow(Task task)
   at Duplicati.Library.Main.Operation.BackupHandler.Run(String[] sources, IFilter filter, CancellationToken token)
   at Duplicati.Library.Main.Controller.<>c__DisplayClass12_0.<Backup>b__0(BackupResults result)
   at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action`1 method)
   at Duplicati.Library.Main.Controller.Backup(String[] inputsources, IFilter filter)
   at Duplicati.Server.Runner.Run(IRunnerData data, Boolean fromQueue)

Edit to add:

Maybe related to this change made in 107?

Replaced WebRequest for WebDAV with HttpClient

DCinME · October 4, 2024, 4:51am

Updated directly from 2.0.9.101 to 2.0.9.108 on a Windows 11 laptop by running the .MSI directly. The process went without any issues.

One thing that I consider strange is the appearance of this message when first navigating to the UI:

It seems redundant to me, as I am used to just being taken to the login screen if I am not logged in. I almost always install Duplicati as a service and so I usually disable the Tray Icon entirely.

Aside from that, everything thus far has worked exactly as expected. I had a look at the new UI and the parts that are done look good except for one thing: The password field is not obscured:

Keep up the good work!

drwtsn32 · October 4, 2024, 5:12am

Yep I noticed that too!

katertje · October 4, 2024, 10:02am

upgraded my machines
macos 12.6 , 2.0.9.106 to 2.0.9.108 no issues, backups are running
macos 15.1, 2.0.9.107 to 2.0.9.108 no issues, backups are running
ubuntu 22.04.1, v107 to v108 no issues, backups are running
ubuntu 20.04.5, v103 to v108, backups are running but…

there is no tray icon, but i can access the web interface using the password
its reporting 2.0.7.103 as a newer version than 2.0.9.108, channel is set to canary

kenkendk · October 4, 2024, 12:13pm

Thanks for reporting that It has been fixed in source already.

That looks like a weird interaction with the timeout code that triggers on the first use. I have not seen it in any of my tests. I just tried a clean install and it did not happen.

Was this immediately after upgrading from 101 to .108?

Is there some kind of error message or other indication of why the icon is missing?

I think this is the “stale update message” error. It happens if there is an update detected, and then you install a new version. It will then keep the message in the queue and show it, even though it no longer makes sense.

Yes, 100% that one. The WebRequest interface is now deprecated in .NET so we are moving away. The check for certificates was previously done via ServicePointManager but now needs to be handled by the HttpClient. I will register an issue for this.

I am aware of a (58!) messages on that topic alone that I need to catch up on. I will do that asap.

Thanks for that ! Yes, we should move into other threads if it is not generally related to the canary builds.

Taomyn · October 4, 2024, 1:42pm

Continues to fail for my Wasabi S3 backup if auto-compact is enabled - when disabled the backup is fine. A manual compact then fails:

 {
  "DeletedFileCount": 0,
  "DownloadedFileCount": 0,
  "UploadedFileCount": 0,
  "DeletedFileSize": 0,
  "DownloadedFileSize": 0,
  "UploadedFileSize": 0,
  "Dryrun": false,
  "VacuumResults": null,
  "MainOperation": "Compact",
  "ParsedResult": "Fatal",
  "Interrupted": false,
  "Version": "2.0.9.108 (2.0.9.108_canary_2024-10-03)",
  "EndTime": "2024-10-04T13:32:54.2559964Z",
  "BeginTime": "2024-10-04T13:29:21.3134095Z",
  "Duration": "00:03:32.9425869",
  "MessagesActualLength": 416,
  "WarningsActualLength": 0,
  "ErrorsActualLength": 1,
  "Messages": [
    "2024-10-04 15:29:21 +02 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Compact has started",
    "2024-10-04 15:29:46 +02 - [Information-Duplicati.Library.Main.Database.LocalDeleteDatabase-CompactReason]: Compacting because there are 99 fully deletable volume(s)",
    "2024-10-04 15:29:46 +02 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  ()",
    "2024-10-04 15:29:48 +02 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed:  (26.94 KB)",
    "2024-10-04 15:29:59 +02 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Started: duplicati-b000b757393b146c594e93bc5319882f1.dblock.zip.aes (49.98 MB)",
    "2024-10-04 15:29:59 +02 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Completed: duplicati-b000b757393b146c594e93bc5319882f1.dblock.zip.aes (49.98 MB)",
    "2024-10-04 15:29:59 +02 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Started: duplicati-i930f7c7345584219bcda0ede2d2a6068.dindex.zip.aes (18.54 KB)",
    "2024-10-04 15:29:59 +02 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Completed: duplicati-i930f7c7345584219bcda0ede2d2a6068.dindex.zip.aes (18.54 KB)",
    "2024-10-04 15:29:59 +02 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Started: duplicati-b04b636e1ecf14c449f39c0863413844b.dblock.zip.aes (49.95 MB)",
    "2024-10-04 15:29:59 +02 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Completed: duplicati-b04b636e1ecf14c449f39c0863413844b.dblock.zip.aes (49.95 MB)",
    "2024-10-04 15:29:59 +02 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Started: duplicati-ie060342596694e6caf962c2f41d508bf.dindex.zip.aes (18.39 KB)",
    "2024-10-04 15:29:59 +02 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Completed: duplicati-ie060342596694e6caf962c2f41d508bf.dindex.zip.aes (18.39 KB)",
    "2024-10-04 15:29:59 +02 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Started: duplicati-b06d665d4f4aa4c7f926b3201774c21fe.dblock.zip.aes (49.96 MB)",
    "2024-10-04 15:29:59 +02 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Completed: duplicati-b06d665d4f4aa4c7f926b3201774c21fe.dblock.zip.aes (49.96 MB)",
    "2024-10-04 15:29:59 +02 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Started: duplicati-ibf658d036cda45d6a6371a5fa2944f38.dindex.zip.aes (18.36 KB)",
    "2024-10-04 15:29:59 +02 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Completed: duplicati-ibf658d036cda45d6a6371a5fa2944f38.dindex.zip.aes (18.36 KB)",
    "2024-10-04 15:29:59 +02 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Started: duplicati-b0a45230a148a41a89da64a2dcb1ab302.dblock.zip.aes (49.93 MB)",
    "2024-10-04 15:29:59 +02 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Completed: duplicati-b0a45230a148a41a89da64a2dcb1ab302.dblock.zip.aes (49.93 MB)",
    "2024-10-04 15:29:59 +02 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Started: duplicati-ibc3e510f43e44efabf830706cc602e64.dindex.zip.aes (18.36 KB)",
    "2024-10-04 15:29:59 +02 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Completed: duplicati-ibc3e510f43e44efabf830706cc602e64.dindex.zip.aes (18.36 KB)"
  ],
  "Warnings": [],
  "Errors": [
    "2024-10-04 15:32:54 +02 - [Error-Duplicati.Library.Main.Controller-FailedOperation]: The operation Compact has failed with error: Data Error\r\nDataErrorException: Data Error"
  ],
  "BackendStatistics": {
    "RemoteCalls": 207,
    "BytesUploaded": 52523930,
    "BytesDownloaded": 157017447,
    "FilesUploaded": 2,
    "FilesDownloaded": 3,
    "FilesDeleted": 201,
    "FoldersCreated": 0,
    "RetryAttempts": 0,
    "UnknownFileSize": 0,
    "UnknownFileCount": 0,
    "KnownFileCount": 27577,
    "KnownFileSize": 719766933845,
    "LastBackupDate": "2024-10-04T11:34:07+02:00",
    "BackupListCount": 16,
    "TotalQuotaSpace": 0,
    "FreeQuotaSpace": 0,
    "AssignedQuotaSpace": -1,
    "ReportedQuotaError": false,
    "ReportedQuotaWarning": false,
    "MainOperation": "Compact",
    "ParsedResult": "Success",
    "Interrupted": false,
    "Version": "2.0.9.108 (2.0.9.108_canary_2024-10-03)",
    "EndTime": "0001-01-01T00:00:00",
    "BeginTime": "2024-10-04T13:29:21.3134166Z",
    "Duration": "00:00:00",
    "MessagesActualLength": 0,
    "WarningsActualLength": 0,
    "ErrorsActualLength": 0,
    "Messages": null,
    "Warnings": null,
    "Errors": null
  }
}

The error in the main backup is:

2024-10-04 15:32:36 +02 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Started: duplicati-b0d0fee3d1f784621834ad1e22e18ec47.dblock.zip.aes (49.92 MB)
2024-10-04 15:32:36 +02 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Completed: duplicati-b0d0fee3d1f784621834ad1e22e18ec47.dblock.zip.aes (49.92 MB)
2024-10-04 15:32:36 +02 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Started: duplicati-icf5a4946c92e4b3eb1c05d80a433fefa.dindex.zip.aes (168.93 KB)
2024-10-04 15:32:36 +02 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Completed: duplicati-icf5a4946c92e4b3eb1c05d80a433fefa.dindex.zip.aes (168.93 KB)
2024-10-04 15:32:54 +02 - [Error-Duplicati.Library.Main.Controller-FailedOperation]: The operation Compact has failed with error: Data Error
SharpCompress.Compressors.LZMA.DataErrorException: Data Error
   at SharpCompress.Compressors.LZMA.LzmaStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at Duplicati.Library.Main.Volumes.BlockVolumeReader.ReadBlock(String hash, Byte[] blockbuffer)
   at Duplicati.Library.Main.Operation.CompactHandler.DoCompact(LocalDeleteDatabase db, Boolean hasVerifiedBackend, IDbTransaction& transaction, BackendManager sharedBackend)
   at Duplicati.Library.Main.Operation.CompactHandler.Run()
   at Duplicati.Library.Main.Controller.<Compact>b__23_0(CompactResults result)
   at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action`1 method)

DCinME · October 4, 2024, 10:44pm

Yeah, it started doing this after the update from .101 to .108. Seems to happen every time I try to load the UI by opening a browser and going to http://localhost:8200.

I noticed a couple of these in the network logs:

ts678 · October 8, 2024, 11:24pm

Default to using .NET built-in Zip code for performance improvements

is my main suspect for an inability to recompress from LZMA. See 2.0.9.107 post for context:

C:\Duplicati\duplicati-2.0.9.108_canary_2024-10-03-win-x64-gui>Duplicati.CommandLine.RecoveryTool.exe recompress zip "C:\tmp\recompress.pre" "C:\tmp\recompress.post"
Listing files on backend: file ...
Found 1 files at remote storage
Found 1 files which belongs to backup with prefix duplicati
1/1: duplicati-b0a69695168d443dfa437b9d44b6de54e.dblock.zip - downloading (773 bytes)... recompressing ... error: System.IO.InvalidDataException: The archive entry was compressed using LZMA and is not supported.
   at System.IO.Compression.ZipArchiveEntry.Open()
   at Duplicati.Library.Compression.ZipCompression.BuiltinZipArchive.OpenRead(String file)
   at Duplicati.Library.Compression.ZipCompression.FileArchiveZip.OpenRead(String file)
   at Duplicati.CommandLine.RecoveryTool.Recompress.Run(List`1 args, Dictionary`2 options, IFilter filter)
Download complete, of 1 remote files, 1 were downloaded with 1 errors
There were errors during recompress of remote backend files!

marceloduplicati · October 10, 2024, 10:16am

There is a [PR](Solves Webdav issue with self signed certificates and accepts appropriate options by marceloduplicati · Pull Request #5588 · duplicati/duplicati · GitHub) addressing this issue, it should be on the next Canary release.

drwtsn32 · October 10, 2024, 2:28pm

Excellent, thank you for the fix!

kenkendk · October 21, 2024, 12:28pm

I have registered this issue:
Improve first-visit experience · Issue #5591 · duplicati/duplicati · GitHub

I think it reflects your experience, but let me know if I misunderstood.

Absolutely right! The built-in .Net zip is much faster than SharpCompress, but only supports Deflate+Store algorithms. In case any one else experiences this, the solution is to request SharpCompress:

--zip-compression-library=SharpCompress

For most cases, there will be an automatic fallback to SharpCompress, but in this case the fallback mechanism is not triggered as .NET was able to open and read the headers without problems.

Taomyn · October 28, 2024, 2:50pm

Was wondering what are the chances this new version could make Duplicati time zone aware? It was quite painful having to adjust all my backups by +1hr ahead of when our clocks changed on Sunday, and I’m sure when the US does the same others will invariably come asking.

ts678 · October 28, 2024, 3:37pm

It sounds like this asks about the issue below (still open)

github.com/duplicati/duplicati

DST changes scheduled backup time

opened 08:38AM - 24 Apr 15 UTC

go-vegan

bug

I've set my backups to run at 11:00 each day, but since the switch to DST, they …run on hour later (12:00). ## <bountysource-plugin> --- Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/11950565-dst-changes-scheduled-backup-time?utm_campaign=plugin&utm_content=tracker%2F4870652&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F4870652&utm_medium=issues&utm_source=github). </bountysource-plugin>

ts678 · October 28, 2024, 3:55pm

Maybe you’re asking about some future version past 2.0.9.108 (which I thought you were on)?

kenkendk · November 4, 2024, 3:22pm

The issue is that the time is stored in UTC, which does not move with DST.
A fix could be to grab the user timezone information from the browser when creating/updating the backup. It is still slightly tricky as we need the actual timezone, not just the offset as that would change.

Taomyn · November 5, 2024, 7:52am

I wouldn’t recommend getting that from the browser as it could be located anywhere and not even be correct, better for Duplicati to get the system time zone itself directly from the machine its running on. Most applications I’ve encountered usually offer an auto setting using the system time zone, or a dropdown list of zones to override it.

Taomyn · November 5, 2024, 8:01am

I have an issue with a backup of one of my Windows machines where it’s been stuck on “Deleting unwanted files” for over a day now. When I look on the server directly, the GUI shows no logging since it deleted a dindex file, task manager and process explorer show no file activity although it does show some open files in the temporary folder which also haven’t changed in the hours I’ve been looking at the issue, plus the database file belonging to the job. The Duplicati-Server.exe process appears to be running as threads come and go, although it’s maxing out one of the CPUs.

The server state in About shows this:

lastPgEvent : {"BackupID":"3","TaskID":24,"BackendAction":"Delete","BackendPath":"duplicati-i2af594563bfc48b0bf257658667bef47.dindex.zip.aes","BackendFileSize":79629,"BackendFileProgress":0,"BackendSpeed":-1,"BackendIsBlocking":false,"CurrentFilename":null,"CurrentFilesize":0,"CurrentFileoffset":0,"CurrentFilecomplete":false,"Phase":"Backup_Delete","OverallProgress":0,"ProcessedFileCount":66882,"ProcessedFileSize":1425580177761,"TotalFileCount":66882,"TotalFileSize":1425580177761,"StillCounting":false}

The dindex file mentioned no longer exists on the remote folder (an SMB share) so it did remove it.

Wondering what I should do. Should I try to stop the job and then rebuild the database?

kenkendk · November 5, 2024, 8:47am

I was considering getting it from the server, but some machines (especially Docker) is usually just reporting UTC.

But I think you are correct that it is better to get it that way and then let the user pick a timezone if it is wrong for whatever reason.

I think I fixed one of the issues with this symptom, but it did not have 100% CPU.

Can you see if it is related to database activity? Is there any disk activity?

If we can narrow down the area that is stuck, that would be useful for reproducing

Otherwise, I would terminate the process and run repair afterwards to clean up any leftover files.