Early preview of .Net8 builds, codename `2.0.8.105`

Releases
1

As announced in the latest canary, the plan is to move to .Net8 builds going forward.
For those that want to help out or try the very latest, I have prepared a set of builds that use the .Net8 build & package process.

This build has a number of fixes reported on the previous builds:

  • Better libicu support
  • Disable Kestrel on port 3001
  • Added libssl for Docker images
  • Fixed some log-spam from SQLite with quoted constants
  • Removed mentions of Mono and updated executable references
  • Fixed SQLite on Apple M3 crashing on log messages
  • Fixed systeminfo (and others) pausing 30s when listing backend options
  • Removed Tardigrade (deprecated 3y ago)
  • Placing MSI install in x64 folder
  • Exclude binaries for Storj due to crashes
  • Re-established AlphaFS and AlphaVSS to Nuget versions

The packages are now signed (Windows + MacOS) and the GPG signatures are included.

Upgrade notice

For Windows users, the new versions of SQLite no longer support the deprecated RC4 encryption. Current builds (.Net4 based) can decrypt the Duplicati-server.sqlite file used by the UI, but the new builds cannot. I have prepared a decrypt tool from the .Net4 line:

The file naming structure:

The filenames are named after operating system, chip architecture and intended use, e.g. linux-x64-gui.
The -gui builds also include the cli executable, but are generally bigger due to the UI libraries.

The suffix denotes the installer type:

  • .zip: Generally just the files, zipped.
  • .deb: Debian package
  • .rpm: RPM package
  • .pkg, .dmg: MacOS packages
  • .msi: Windows Installer

Docker images:

Docker images are pushed with the tags 2.0.8.105 and Debug.

The armv7, amd64 and arm64 architectures are included.

List of images

1 Like
2

Looking better. Thanks! To expand on a prior issue, here’s the system tray expanded showing an error:

image

The black on gray is slightly easier to read than black on black. Shouldn’t the red area be larger though?

Beyond that, VSS seems to be good, and my actions that used to crash and log app events are working.

The Export As Command-line now point to a .exe rather than a .dll. Good.

1 Like
3

Testing results for duplicati-2.0.8.105_debug_2024-04-16-win-x64-gui.msi on Windows 11 Pro x64.

Weird results. If you install it over an existing install of 2.0.7.1 beta, nearly all of the executables are missing afterwards. Duplicati.WindowsService.exe, Duplicati.CommandLine.exe, etc.

I have verified that none of the programs are in use or would have anything impeding them being appropriate dealt with. On a clean install this problem does not occur.

Before (2.0.7.1 beta):

image
image706Ă—308 5.93 KB

After (2.0.8.105 debug):

image
image814Ă—148 3.32 KB

Edit:
Aside from the above, most of the problems I encountered with 2.0.8.104 have been resolved. Duplicati now correctly installs to C:\Program Files instead of C:\Program Files (x86). VSS is working again. The problem with the Storj dll are no longer present.

It might be me imagining things, but these newer builds of Duplicati seem to be much faster than the older .NET4 builds, though I’m just going by seat-of-the-pants measurement. I’ll have to let my logs capture more data before I can say conclusively.

I still find the database decryption process to be somewhat finicky. It sometimes throws errors for no reason that I can discern, even when I have gone in and clicked the “Unblock” attribute.

4

I think a re-design of the tray-icon is needed, but that deserves its own issue.

Wow! That is unexpected. Only thing that I can imagine causes something like this is an incorrect upgrade code in the MSI. I will try to set up a reproduction of this issue.

At least the startup should be much faster, as all the updater checks are no longer there. Apart from that there are many performance optimizations with each .NET iteration and this is skipping 4 versions :smiley:

I can make a signed version, but I think it would perhaps have to be an MSI package for it to escape the “Unblock” problem.

Before beta release there should be some automated way to update the database.

5

My test was different. I think I just installed x64 over the previous .msi install which wrongly went into (x86). Reasonable result was empty C:\Program Files (x86)\Duplicati 2, and a well-populated usual spot.

Change Wix installer to after-installfinalize #3886 changed some orderings awhile ago, and I don’t recall if change is responsible for my increased confidence about over-install without doing an uninstall before that. Version downgrades have also caused issues from time to time, though I couldn’t spot such things here…

That’s about all I can say about history, for whatever it’s worth. Certainly a good repro will find some things.

6

I can reproduce this issue :cry:

That was very important information! I will try to build the package with this reverted, and hope it is not the beta MSI that is causing the problem.

7

Looks like the beta MSI is the problem.

The normal install has 517 files, and an install with on a machine with the beta already present results in 417 files. Looking at the files, it appears that most files that have the same name in both MSI files are removed as part of the replace process.

8

Note that it was also meant to fix a problem (follow the mentions from the PR). The release note said:

Release: 2.0.4.26 (canary) 2019-09-02

had more discussion, including the talk of in-place upgrade I mentioned, and you on CoCoL versions.

9

It looks very much like all the smart features in MSI are standing in the way of simpler “xcopy install” methods :confused:

Anyway, I managed to “fix” the problem by rolling back that change to the MSI.
I think there was a problem with the version number in the MSI and that somehow triggered the same issue.

The latest builds I have fixes the icon and correctly replaces the files as far as I can see.

10

On a different topic, I did a .deb. install, systemctl couldn’t find the unit file, and I’m trying to troubleshoot. Duplicati looks like it puts the unit in /lib/systemd/system/duplicati.service. On this Linux Mint 20, there is /usr/lib which is not the same (via symlink) as /lib, and systemd also has unit files (or symlinks) in /etc. Expertise in systemd would help here.

EDIT 1:

Comparing what I have on the system now. If need be, I can try some other releases top see what they do:

$ dpkg --contents duplicati_2.0.6.1-1_all.deb | fgrep duplicati.service
-rw-r--r-- root/root       245 2021-05-03 02:49 ./lib/systemd/system/duplicati.service
-rw-r--r-- root/root       245 2021-05-03 02:49 ./usr/lib/duplicati/debian/duplicati.service
$ dpkg --contents 2.0.8.100_debug_2024-04-04-linux-x64-gui.deb | fgrep duplicati.service
$ dpkg --contents 2.0.8.101_debug_2024-04-04-linux-x64-gui.deb | fgrep duplicati.service
$ dpkg --contents duplicati-2.0.8.104_debug_2024-04-11-linux-x64-gui.deb | fgrep duplicati.service
-rw-r--r-- root/root       245 2024-03-06 09:11 ./lib/systemd/system/duplicati.service
$ dpkg --contents duplicati-2.0.8.105_debug_2024-04-16-linux-x64-gui.deb | fgrep duplicati.service
-rw-r--r-- root/root       245 2024-03-06 09:11 ./lib/systemd/system/duplicati.service
$

EDIT 2:

Doing sysctemctl status duplicati fails, systemctl enable duplicati fixes that, systemctl disable duplicati still allows systemctl status duplicati to work. The enable and disable seem to make the /etc/systemd symlink.

# systemctl status duplicati
Unit duplicati.service could not be found.
# systemctl enable duplicati
# systemctl status duplicati
â—Ź duplicati.service - Duplicati web-server
     Loaded: loaded (/lib/systemd/system/duplicati.service; enabled; vendor preset: enabled)
     Active: inactive (dead)
# systemctl disable duplicati
Removed /etc/systemd/system/multi-user.target.wants/duplicati.service.
# systemctl status duplicati
â—Ź duplicati.service - Duplicati web-server
     Loaded: loaded (/lib/systemd/system/duplicati.service; disabled; vendor preset: enabled)
     Active: inactive (dead)
# systemctl enable duplicati
Created symlink /etc/systemd/system/multi-user.target.wants/duplicati.service → /lib/systemd/system/duplicati.service.
11

It seems to have trouble popping up messages, testing on the default Firefox on Linux Mint 20. Originally, I thought it was ignoring the Save button on job creation, but after installing chromium to test same config, it complained about my weak password upon Save. Firefox also had trouble finishing Test connection UI.

I haven’t tried a similar test on Windows yet.

EDIT 1:

Other than that, a very basic backup-to-folder and restore (even using no-local-blocks) seems working.

I was running /usr/lib/duplicati/Duplicati.GUI.TrayIcon, but I can’t find even a trying-to-hide icon.

EDIT 2:

Trying a .zip install on Windows next, basically the way I install a Canary (I have a tree of Duplicati’s here).
One difference is usually the .zip has the files. Now the .zip has the folder again, and then files inside that.

EDIT 3:

Firefox 124.0.2 on Windows lacks the UI glitch of same version on Linux Mint 20. Besides Test connection, another easy test is to start creating a new backup, jump right to Save, see if it pops up a complaint or not.

EDIT 4:

UploadSpeed for larger (50 MB default, or close) dblock files is far slower than before, roughly 200 KB/sec, compared to maybe 6 MB/sec on 2.0.7.102 Canary going to Backblaze B2 over a fast link (limited by WiFi). DownloadSpeed is closer, maybe 3 MB/sec compared to 12 MB/sec on the older version. Testing on same Windows 10 system with same config and with no job or GUI throttling set. I just moved the old config over.

EDIT 5:

I’m not sure if I’ve seen this weirdness before, but B2 doesn’t usually decide to 503 right at the backup end:

2024-04-18 19:15:51 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-icc15aa7e025149be87df1600de9d0340.dindex.zip.aes (26.68 KB)
2024-04-18 19:15:51 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 26.68 KB in 00:00:00.4901573, 54.44 KB/s
2024-04-18 19:15:51 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-icc15aa7e025149be87df1600de9d0340.dindex.zip.aes (26.68 KB)
2024-04-18 19:15:51 -04 - [Profiling-Timer.Begin-Duplicati.Library.Main.Operation.Common.DatabaseCommon-CommitTransactionAsync]: Starting - CommitAfterUpload
2024-04-18 19:15:51 -04 - [Profiling-Timer.Finished-Duplicati.Library.Main.Operation.Common.DatabaseCommon-CommitTransactionAsync]: CommitAfterUpload took 0:00:00:00.190
2024-04-18 19:15:51 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-20240418T231133Z.dlist.zip.aes (939.14 KB)
2024-04-18 19:15:56 -04 - [Retry-Duplicati.Library.Main.Operation.Backup.BackendUploader-RetryPut]: Operation Put with file duplicati-20240418T231133Z.dlist.zip.aes attempt 1 of 1 failed with message: 503 - service_unavailable: no tomes available
System.Exception: 503 - service_unavailable: no tomes available
   at Duplicati.Library.Backend.Backblaze.B2AuthHelper.AttemptParseAndThrowException(Exception ex)
   at Duplicati.Library.Backend.Backblaze.B2AuthHelper.ParseException(Exception ex)
   at Duplicati.Library.JSONWebHelper.GetResponseAsync(AsyncHttpRequest req, CancellationToken cancelToken, Object requestdata)
   at Duplicati.Library.JSONWebHelper.ReadJSONResponseAsync[T](AsyncHttpRequest req, CancellationToken cancelToken, Object requestdata)
   at Duplicati.Library.JSONWebHelper.GetJSONDataAsync[T](String url, CancellationToken cancelToken, Action`1 setup, Func`3 setupbodyreq)
   at Duplicati.Library.Backend.Backblaze.B2.PutAsync(String remotename, Stream stream, CancellationToken cancelToken)
   at Duplicati.Library.Main.Operation.Backup.BackendUploader.DoPut(FileEntryItem item, IBackend backend, CancellationToken cancelToken)
   at Duplicati.Library.Main.Operation.Backup.BackendUploader.<>c__DisplayClass20_0.<<UploadFileAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Duplicati.Library.Main.Operation.Backup.BackendUploader.DoWithRetry(Func`1 method, FileEntryItem item, Worker worker, CancellationToken cancelToken)
2024-04-18 19:15:56 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Failed: duplicati-20240418T231133Z.dlist.zip.aes (939.14 KB)
2024-04-18 19:15:56 -04 - [Error-Duplicati.Library.Main.Operation.BackupHandler-FatalError]: Fatal error
System.Exception: 503 - service_unavailable: no tomes available
   at CoCoL.AutomationExtensions.RunTask[T](T channels, Func`2 method, Boolean catchRetiredExceptions)
   at Duplicati.Library.Main.Operation.BackupHandler.FlushBackend(BackupResults result, IWriteChannel`1 uploadtarget, Task uploader)
   at Duplicati.Library.Main.Operation.BackupHandler.RunAsync(String[] sources, IFilter filter, CancellationToken token)
2024-04-18 19:15:56 -04 - [Profiling-Timer.Begin-Duplicati.Library.Main.Database.ExtensionMethods-ExecuteNonQuery]: Starting - ExecuteNonQuery: PRAGMA optimize
2024-04-18 19:15:56 -04 - [Profiling-Timer.Finished-Duplicati.Library.Main.Database.ExtensionMethods-ExecuteNonQuery]: ExecuteNonQuery: PRAGMA optimize took 0:00:00:00.000
2024-04-18 19:15:56 -04 - [Profiling-Timer.Finished-Duplicati.Library.Main.Controller-RunBackup]: Running Backup took 0:00:04:23.409
2024-04-18 19:15:56 -04 - [Verbose-Duplicati.Library.SQLiteHelper.SQLiteLoader-CustomSQLiteOption]: Setting custom SQLite option 'cache_size=-200000'.
2024-04-18 19:15:56 -04 - [Profiling-Timer.Begin-Duplicati.Library.Main.Database.ExtensionMethods-ExecuteReader]: Starting - ExecuteReader: SELECT "ID", "Timestamp" FROM "Operation" ORDER BY "Timestamp" DESC LIMIT 1
2024-04-18 19:15:56 -04 - [Profiling-Timer.Finished-Duplicati.Library.Main.Database.ExtensionMethods-ExecuteReader]: ExecuteReader: SELECT "ID", "Timestamp" FROM "Operation" ORDER BY "Timestamp" DESC LIMIT 1 took 0:00:00:00.000
2024-04-18 19:15:56 -04 - [Profiling-Timer.Begin-Duplicati.Library.Main.Database.ExtensionMethods-ExecuteNonQuery]: Starting - ExecuteNonQuery: PRAGMA optimize
2024-04-18 19:15:56 -04 - [Profiling-Timer.Finished-Duplicati.Library.Main.Database.ExtensionMethods-ExecuteNonQuery]: ExecuteNonQuery: PRAGMA optimize took 0:00:00:00.000
2024-04-18 19:28:36 -04 - [Verbose-Duplicati.Library.Modules.Builtin.SendHttpMessage-HttpResponseMessage]: HTTP Response: 200 - OK: 
2024-04-18 19:28:38 -04 - [Profiling-Duplicati.Library.Modules.Builtin.SendMail-SendMailResult]: Whole SMTP communication: Connected to smtp://smtp.gmail.com:587/?starttls=always

During the odd 13 minute gap in action, the status bar had little to say – I think it said Error at right-- and the miniature colored spot on the tray icon was green. Now that the error is noticed, icon spot is red and I have expected big red popup in the web UI. Server log error was dated 7:28. Job log is logging the failure at 7:15.

503 from Backblaze is kind of an expected occasional thing per their architecture, and setting number-of-retries=0 lets me use it to simulate big network failures that actually exhaust the default number of retries.

Duplicati’s handling of the situation was kind of weird, and I’m not sure I’ve seen this on previous releases, however B2 usually does this on the first upload of the backup/ This time, though, it did it on its ending dlist.

Maybe someday a Duplicati test person will make some nice tool to put network errors at desired places…

12

I did some command line test restores of a small backup using 2.0.8.105. I also ran the same test restores using 2.0.7.1 to make sure the outputs were the same. 2.0.8.105 is noticeably faster.

Backup size: 7.28 GB in 33263 files. Block size is the default 102400 bytes. Volume size 100MB.
Restored from a SMB NAS over 1Gbps local network to a USB-attached HDD. No-local-blocks was enabled to simulate restoration from nothing. I ran each test three times.

2.0.7.1
Restore 1: 33 minutes, 40 seconds
Restore 2: 33 minutes, 10 seconds
Restore 3: 33 minutes, 27 seconds

2.0.8.105
Restore 1: 27 minutes, 55 seconds
Restore 2: 26 minutes, 57 seconds
Restore 3: 28 minutes, 17 seconds

I also make frequent use of the WebDAV and OneDrive v2 backends. I intend to test them too when I get an opportunity.

13

I tested these on Debian+Ubuntu and systemctl seemed to recognize them there.
Since Mint is based on Ubuntu, I don’t see why it should be different.

I think this is how systemctl is supposed to work. If you have not enabled it, it is not known.
Maybe Duplicati should default-activate, but that would be another issue.

I guess these issues will keep showing up until we upgrade from AngularJS.

That seems to be an issue on Mint 20. On Mint 21.3 Cinnamon, I get “an icon”, but obviously not the one I want:

Screenshot 2024-04-19 at 14.27.30
Screenshot 2024-04-19 at 14.27.301770Ă—1350 168 KB

I think that is how it “should be” otherwise you will potentially bomb the Download folder if people extract there. On the other hand, it is zip program dependent how they treat folders like that, so there is no magic solution. At least inside a folder, you can move the folder if you like.

There is some support for this, but not nearly as great as it should be. Anyway, I think it is a task after the .NET8 update.

All the more reason to switch :slight_smile:

The internals of the webrequests have changed significantly after .NET4, but I would be surprised if it has any impact on speed, as that is (almost always) limited by network speed anyway.

14

The OneDrive v2 backend (and the other MS Graph based ones, including SharePoint v2) have support for both the old WebRequest infra that most other backends use as well as the newer HttpClient stack (I’m not sure if any other backends have used the HttpClient one in Duplicati since it was added). The HttpClient version should be the default on non-Mono environments - though I’m not sure if I’ve seen anyone compare speeds between them.

@DCinME - I wrote the initial versions of the OneDrive v2 backend, so am curious to hear back what you find in your testing.

15

From my investigations inside WebRequest in .NET7, I have found that all calls ends up in the HttpClient code. There is simply a wrapper around the deprecated WebRequest that mimics the old interface but calls HttpClient.

So I would expect exactly much the same performance from either, but with a minor additional memory overhead for using WebRequest (and a ton of build warnings).

1 Like
16

@kenkendk pretty much nailed it. My upload speed would mask any detectable difference in performance between the old and the new when using OneDrive v2. However aside from that I’m happy to report that it appears to be working great under .NET8. In my limited testing thus far I have experienced no issues.

In fact, I’ve only ever experienced two issues with it. In the recent past there was the “RetryAfterWait” problem that would cause Duplicati to get throttled, and the other is Duplicati sometimes reports a false positive that I’m running out of space on OneDrive. In reality I’ve got plenty of space, but it’s counting things in the Recycle and Secondary Recycle bins towards my quota when it checks. A minor nuisance that doesn’t come up very often.

I use it every day, thanks for writing it @tygill.

17

I did not understand this correctly, but looking at the previous builds, I see the issue.
I copied in one set of icons for all operating systems when I removed all the old frameworks.
I have found the correct version of the icons and re-added them.