Hi, I just got Duplicati setup on my TrueNAS NAS as a jail. I started an initial backup of one of the smaller datasets on my NAS and I am experiencing some worrying performance.
System details:
Intel(R) Xeon(R) CPU X5660 @ 2.80GHz x2
Disk config: raidz50 9 disks 3 disks per vdev
Network: 10GbE
Dataset: ~300GB
Currently I am getting 5MB/s speed according to duplicati.
It seems like most of the time is spent at this step: Backup_ProcessingFiles once it gets to the transfer step it is super fast because there is plenty of bandwidth available.
Is this just the limit of my CPU and duplicati not being multithreaded? I have 40TB to backup so I need to see where the bottleneck is.
For the main issue, it looks like you’re doing a great job with SQL improvements. Much appreciated!
As a total side issue, we’re currently talking about what to do for Duplicati on FreeBSD in the future.
Linux-based TrueNAS SCALE is now out (very recently), with a migration path from TrueNAS CORE. Seemingly both will be supported for awhile. TrueNAS CORE is much more of a “known” thing now…
Meanwhile, .NET Framework and mono are functionally stagnant, as Microsoft pursues .NET 5/6, etc.
Possibly our FreeBSD users who stick with it will just not enjoy the dotnet future that will be forced by things such as language and library usages, until dotnet on FreeBSD happens (it’s being worked on).
Do you have thoughts on this? Small sample size, but you’re the most recent forum FreeBSD poster.
Any others who wander by, feel free to join. There’s also an open pull request for those using GitHub.
While not ideal or as user friendly as the TrueNAS Jail plugin. TrueNAS core does support VM’s. I don’t know what the performance is like with them. So that is an alternative. I can probably test that and see if it is at all viable performance wise for the type of datasets that would be on TrueNAS.
I usually think of them as kind of isolated, but I might just not be expert enough to expose the host files.
This issue reminds me a bit of the Docker problem, but it seems Docker on FreeBSD is broken for now. Linuxulator appears to be doing better, but it’s still somewhat behind and has limited application testing.
Duplicati will need some time to migrate onto the newer .NET. Meanwhile TrueNAS SCALE will mature.
I don’t expect an entire user base to be willing to jump into either instantly. Meanwhile old stuff remains.
As legacy software, old Duplicati maintenance may slow, but honestly it’s slow now (lack of volunteers).
One irritating thing for decision-making is I’m unsure we know how many FreeBSD users Duplicati has. https://usage-reporter.duplicati.com/ shows Windows, OSX, and Linux, but I suspect FreeBSD is Linux:
The possible good news is there’s not just a type, but a version, and I don’t know what clues that has:
Sometimes usage reports remain in tmp folder. If you see a dupl-usagereport-*.json, please read.
You might find some other files beginning with dup-, but those would be another sort of temporary file.
Thanks for your help in figuring out how to handle FreeBSD.
Ya, looks like it would just use the internal network routing in the OS to access a network share on the TrueNAS host. So performance probably would be good enough.
Thanks for the look. I’m making inquiries on whether the usage reporter database holds osversion field. clrversion at 5.10 was sort of a happy accident with Duplicati’s needs lining up with FreeBSD’s latest.
And we can presumably add a VM (maybe running Linux to avoid Windows licensing?) as another path.