I have a Windows 10 VM where I have added a 2nd NIC (a wireless USB dongle) to offload network traffic from the virtual NIC. However, if I use the 2nd NIC for Duplicati backups the verification step fails with “The decryption operation failed, see inner exception.”, and the inner exception is “The specified data could not be decrypted”.
I have looked over the options and don’t see anything that seems relevant. Is there something I can do to make Duplicati “happy” with the new NIC short of creating a completely new backup?
2.0.7.103_canary_2024-04-19, Windows 10 VM running under KVM/QEMU on a Fedora 39 host, backing up to b2.
Why do you think that would help? You could certainly make a small new test backup if you like.
Possibly it will fail in the usual way, but there’s also a lot of testing previously suggested for this:
Since the Microsoft .NET code might have changed over the years, you can try installing the Release: 2.0.9.107 (Canary) 2024-09-11 or newer release that runs on .NET 8 instead of the somewhat stagnant-for-years .NET Framework that 2.0.8.1 Beta and earlier use on Windows.
Or possibly the bug is in Windows. You can do your own search on your error. I think it’s TLS decryption, which would be Windows, however .NET might be over-represented in reporting. Sometimes it’s not clear what language some program is made in. BTW, PowerShell is .NET.
Since you’re already on a Canary, you probably know that they’re testing releases, however a method to limit risks (2.0.9.x has had a lot of changes), is to install from .zip to some unusual folder to run CLI connectivity tests as shown previously. Don’t run things involving the jobs, as possibly you don’t want the database update that makes it harder to go back (if you wish that).
Having an installation in some other folder, and running limited commands, won’t change DB:
My assumption is that there is “something” within the backup config that is “tied” to something with the NIC config. And I stress, assumption. From there it follows that creating a new backup config using the new NIC would work - for the new NIC.
Re: upgrading to the latest Canary - I plan to do so when I have the time to give it proper attention. If there is no “breaking change” to the files on b2, I could snapshot the VM’s virtual disk, do the upgrade and test. On a snapshot rollback (if needed) the only issue would then be files on b2 that Duplicati wouldn’t recognize - I assume (more assumptions!) that there would be a way to get Duplicati to automatically clean those up.
My assumption is that the Duplicati config is far, far away from the NIC. Windows owns the NIC. Additionally, .NET is in between. Duplicati is a high level application, not handling any hardware.
That sounds good. I just wanted to suggest an early way to test with it before jumping all-in.
The destination format hasn’t changed. The Duplicati-server.sqlite server database changed.
This makes it somewhat more awkward than usual to revert to old Duplicati, if one wishes to.
More findings:
The NativeErrorCode seems to have gotten split in copying. If I go to Windows error Decoder https://james.darpinian.com/decoder/, -2146893008 search is giving me this for a description:
SEC_E_DECRYPT_FAILURE: The specified data could not be decrypted.
Plug that hex value into Windows calculator with QWORD length, get
So this Windows error in its SChannel (a.k.a. Secure Channel) appears to be the start of issue. Sometimes the SChannel user can have an impact, but for Duplicati, the user is the .NET code.
I saw some say that NOT using TLS 1.3 avoids issue, but I’m not sure your Windows 10 does it. allowed-ssl-versions can be tried, but I’m not sure it can be added to the test programs I named.
As I put in my other post, I expect a change in NIC - or network config in general - to be transparent to a client program (server programs are another matter). But in this case something is obviously not transparent.
Which makes me wonder if somehow Windows 10 has changed that NIC from a “private” network to a “public” network, or similar? No idea if it would even matter. I’ll have to check…
I can also try changing allowed TLS versions.
I will post what I find when I have another chance to boot that VM.
This seems like it would result in a more solid failure to communicate, yet your backup is
which is after (generally) a list at start of backup, and a bunch of put (upload) operations.
This is why I’m asking you to test those. The performance test is to see if perhaps the new networking is losing performance due to some packet damage such as delay, loss, or swap.
The NIC generally just transports packets and has nothing to do with routing, firewalling, etc.
Another way to test packet transport (somewhat) is with ping (but probably not excessively):
C:\>ping api.backblazeb2.com
Pinging api.backblazeb2.com [104.153.233.180] with 32 bytes of data:
Reply from 104.153.233.180: bytes=32 time=76ms TTL=52
Reply from 104.153.233.180: bytes=32 time=77ms TTL=52
Reply from 104.153.233.180: bytes=32 time=79ms TTL=52
Reply from 104.153.233.180: bytes=32 time=79ms TTL=52
Ping statistics for 104.153.233.180:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 76ms, Maximum = 79ms, Average = 77ms
C:\>
Call the Native API is the citation for that host. B2 starts there, then other hosts are used later.
is Duplicati’s B2 authorization. It’s doing high level HTTP requests. All of the details are below.
Network interface controller describes how some NICs have support for higher levels like TCP. Generally I think the NIC leaves TCP to the OS. Your use has virtualization added to the stack. Possibly you can also try reconfiguring that with new NIC. Default setup looks to me to be NAT.
The error occurs during the post-backup verify phase. Duplicati considers the backup to be good, but reports 3 errors. And it goes through all the retries, etc., so the error is persistent, not behaving like a network performance issue or anything transient, especially considering that it just completed a “good” backup. (Backup was after Windows Update, so there was definitely backup activity.)
I.e., the “‘list’ at start of backup, and a bunch of ‘put’ (upload) operations” have already been performed as part of the backup, if I am understanding your suggestion correctly.
Now that we’re in agreement, the next steps (to me) would be the other testing I’m suggesting, however that might just confirm what we suspect is going on: somehow the get operation fails.
Have you ever tested Restore? If you can’t download, restore might have the exact same error.
I’m still seeing reports that this is TLS 1,3, but Windows 10 shouldn’t have it on unless you did it (which looks pretty cumbersome, involving editing the registry). Windows 11 has it on by default.
I’m not sure how solvable this will be, but it’s certainly possible to test more and change network configuration below Duplicati. I can’t advise in detail on your virtualization setup, but for example:
As per your other post the first, original, NIC is a virto NAT, and the 2nd is a USB device passed-through to the VM.
Re: TLS - I don’t recall changing anything at the Windows level. If I changed anything at the Duplicati level it would have been to fix/work-around an issue. I would have to check to confirm.
Based on your research I’m thinking the best path forward is to do the Canary upgrade.
It’s certainly worth a try, plus this will become the regular (non-Canary) Duplicati someday.
I’m discouraged by the dotnet devs’ comments on GitHub, but maybe something changed.
I’m curious if both of your NICs in the VM have a default gateway set. This causes communication problems in Windows (and perhaps other operating systems) because it will inconsistently pick among the gateways you have defined.
If this is the case, one solution might be to remove the default gateway from your virtual NIC so that only local subnet traffic is sent through that NIC. This would ensure off-subnet (like internet-bound) traffic is sent out the wireless USB NIC.
Thank you for the suggestion. Only one NIC is active at a time, and there are no other network-related issues, so I don’t think there is a gateway issue.
If Windows shows the inactive NIC as disabled/disconnected, then yeah the multiple gateway thing shouldn’t be an issue for you.
I agree with previous comments in this thread that Duplicati is a high level application and isn’t involved with anything down at the network interface level. That’s on the operating system.
I might suspect a buggy network driver, faulty network hardware, or maybe issue with the USB passhtrough. But if any of those were the case, I’d think you’d have issues in other programs too, not just Duplicati.
It is puzzling.
Running Wireshark inside the VM might provide some insight if the issue is at the TCP or lower layer.
After upgrading to “2.0.9.107 (Canary) 2024-09-11” the problem persists. The only change is that with this release the GUI is no longer providing the text for the “inner exception”, so without doing something special with logging I can’t see the details.
At any rate, I see that “2.0.9.108_canary_2024-10-03” has been released. I tried to download but it can’t find the file at the moment (?); as soon as I can download and install I will try with that release.
To be clear, I can backup without errors using either NIC. The virtio NIC has no errors, the USB Wi-Fi NIC only has errors during the final verification pass, but a test restore succeeds. So this a manageable situation for me, and more about seeing if it is possible to identify the issue and possible solution for anyone else that might encounter it.
I really doubt this is a duplicati issue. It doesn’t have anything to do with lower level networking.
Out of curiosity what is the hardware model and driver version you are using? What do you think about analyzing traffic with Wireshark?
Another idea: instead of passing through the USB device directly, have the host hypervisor manage the hardware and then just use a virtual NIC inside the VM. If it is a guest driver issue, this might be a workaround.
