Hi there. I’m trying to complete my first backup using Duplicati, however it’s very slow.
I’m backing up to my unraid server over the gigabit lan and am seeing a constant speed of 8mbps. Even without cache I can write to the array at 40mbps. Why is it so slow? Is there anything I can change?
Welp, I guess it just sucks then, and can’t complete a simple backup.
Seeing with what? Task Manager? Is this Windows, Linux, something else? What is drive performance? Typically the need to do the compression and encryption in temp folder will prevent getting to LAN speed.
Constant speed is a little unusual. Usually it’s a little bursty, but that might depend on protocol. Is it SMB?
What do the performance tools say is busy? The drive-is commonly a limit (still seems slow…) but needs measuring. On extremely slow systems (more often an underpowered NAS than a PC), the CPU can limit.
Thanks for your reply.
Seeing this with Duplicati’s interface.
The machine to be backed up is Win 10. Backing up to Unraid.
On the local machine, at the worst case there’s some spinning rust @120MBps, best case several nvme with speeds up to 3000MBps. On the array write speed is 40MBps as I’m skipping the cache.
According to the Duplicati interface it’s very constant. Between 7.8 and 8MBps. Yes, protocall is SMB.
Do you mean built in performance tools on the host and target machines? Neither of them are busy, and neither of them lack for recourses. CPU on the backup machine is R9 5900X, on Unraid it’s i5 4590. The config of Duplicati is completely default, except that encryption is turned off.
Thanks!
Mechanical drives are far better at sequential than random, and Duplicati performance that has some involved is probably somewhere in between. Below is the computer I’m on. Note the huge differences.
Probably host e.g. Task Manager and Resource Monitor, and maybe mostly disk but you’d have to look.
The target probably has an easier time (if it’s disk limited) because it’s sequential writes, and you have seemingly measured that already. The source system scans for changed source files by examining file timestamps, then changed files must be read to find the changed blocks, which then go into a .zip file created in the user’s Temp folder. Actions are recorded in a database typically in user profile. When file reaches configured remote volume size, it is encrypted and copied out. All this disk activity is in parallel.
See backup logs for statistics on what there is, and what was modified (which gets a file read-through):
Your database location is on the Database screen, but is probably in the user profile, as Temp might be. tempdir option can move the folder Duplicati uses for temporary files. For source, I guess it’s where it is however because this is Windows you can use usn-policy to use the NTFS journal to avoid full search. Doing so requires an Administrator account with elevation in effect (e.g. a UAC prompt). SYSTEM will of course work (and doesn’t prompt you), but typically needs setting up Duplicati as a service (extra steps).
If somehow Duplicati is bogged down on its SMB transfer (not my first guess), you can backup to a local folder to see how fast that goes. Is this the initial backup or a later one? Initial backup disk use is probably mostly affected by block processing and not by searching for changes because everything needs backup.
I’m picking on the drive because in my Task Manager I often see it fully utilized and below 8 mbits/second.
I’m not sure of the exact algorithm that produces status bar speed. Bursty speed should be visible in Task Manager on a network interface (assuming it’s not drowned out by other network activities). One can also see transfer speed very clearly in logs at Profiling level. This run is upload speed limited. Yours likely isn’t.
2021-05-14 13:43:04 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 49.97 MB in 00:01:43.3480785, 495.14 KB/s
Profiling logs are huge. For just speeds, you can use log-file with log-file-log-filter and not log-file-log-level.
Getting a rougher idea of upload speeds can be done at Information or Retry level, even in About → Show log → Live. It will be a little confusing because there are parallel uploads to try to more fully fill the network.
--asynchronous-concurrent-upload-limit (Integer): The number of concurrent
uploads allowed
When performing asynchronous uploads, the maximum number of concurrent
uploads allowed. Set to zero to disable the limit.
* default value: 4
If you cut that to 1 it should start looking sequential, with a dblock going out (default 50 MB) then its dindex.
There is a race between production of volumes to upload, and upload rate, If you want to view your Temp folder you can see things either backing up there (if production is faster than upload) or looking a bit empty.
Thanks for the detailed reply, I appreciate the time you put in.
That being said, I’m not going to pull my hair out over it. Several other backup utilities can write to the array at full speed, duplicati is the odd man out, so I’ll just use one of them instead.
It seems it’s for something other that what I was trying to use it for.
Obviously use whatever works for you, but note that the heavy processing does give benefit of storing only changed data in any given backup. It’s not by any means a file copy, so be careful what you compare with.
High CPU usage while backing up #2563 is an issue that just got more posts. One user went to restic to solve a CPU load issue (and I think its GUI is lacking, but that does do the upload-only-changed-data plan).
Big Comparison - Borg vs Restic vs Arq 5 vs Duplicacy vs Duplicati is one of several forum comparisons, and one where Duplicati seemed to be pretty fast. Yours is unusually slow, and the bottleneck is not clear.
Specifics are still vague, but if you have some other satisfactory solutions all set, there’s no need to push.
Well, I’m certainly not comparing to a file copy. As far as I’m aware, block level, incremental backups are table stakes for a modern backup solution.
For completeness, my processor sits idle through all of this.
I have the same issue here, very slow backup speed to my nas on gigabit lan, using samba.
Coping files to the backup directory manually happens with normal speed. But during backup I got 1.5MB/s max.
iotop on linux gives me this
919 be/3 root 0.00 B 24.00 K 0.00 % 98.05 % [jbd2/dm-21-8]
I know this is some kind of ext4 journaling event, but it makes no sense, hence the 1.5MB/s speed. The storage drives are 2x Toshiba P300 with a single md raid1 lvm volume. The duplicati 2 host machine is Windows 11, but I got the same problem with Windows 10.
On both Linux and Windows I have/had normal speed LAN speed so it shouldn’t be the OS.
By default, the code searches and checks the files, puts them into container files of max size, and while continuing on that it uploads them as it maxes out each container file. eg a 50MB container file should upload faster than 1.5MB/s.
If you watch the backup files on the remote drive as the backup is running, you might see something else happening than you are expecting. I’m not sure.
Could also be samba related or something there I suppose since manual copy may include different code than the application uses.
If it is Duplicati and also not user error, the closest I’ve seen is only for a false speed rundown from a code loop issue which ends with a frozen UI.
So this needs to be narrowed down further.
Another thing, you could try using ssh instead if the NAS allows it. It would be less security risky than the local machine having direct access to the backup files as a shared drive via samba sharing as well. ssh can be used with Windows as well.
I’m curious to know if adding and enabling the advanced option, “use-block-cache” changes the results at all?
I’m curious to know what Task Manager and Resource Monitor say is going on with disk and CPU.
Disk is probably less of a worry if drive is an SSD, but preparing backup files is very I/O-intensive.
@Xavron described. You can look in your %TEMP% folder to watch default 50 MB files going by.
If there is no queue of big dup-* files, it’s a production limit (@JimboJones idea might change it).
If there are several in the queue (configurable by asynchronous-upload-limit) it’s limited by upload.
I’m assuming that the iotop is from the NAS, but please mention if there’s Duplicati-on-Linux too.
If you want to test some Duplicati code (although not the backup code), you can copy files somewhere (preferably not to the backup destination – use a different folder) starting with Export As Command-line. Edit the Target URL and use it with Duplicati.CommandLine.BackendTool.exe to manually time file put.
Duplicati times all of its backup file put operations, but you need a Profiling log or log-file-filter to look at:
2021-03-10 09:44:16 -05 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 49.97 MB in 00:01:45.9147070, 483.12 KB/s
(example from my remote backup where speed is limited by upload link – I wonder what your limiter is?)
Measured how?
Please read this post
There are a lot of other points in this topic that aren’t being answered by the poster (but should be). Comparing Duplicati to a different method like in EaseUS that AFAIK doesn’t deduplicate is not fair.
Features (from Introduction in Duplicati Manual)
Deduplication
Duplicati analyzes the content of the files and stores data blocks. Due to that, Duplicati will find duplicate files and similar content and store them only once in the backup. As Duplicati analyzes the content of files it can handle situations very well if files and folders are moved or renamed. As the content does not change, the next backup will be tiny.
Sorry it takes time, and you have the long article on possible ideas, but data integrity may come first.
IMO this is more important to getting out of Beta, and more performance comes after better reliability.
Someday (volunteers permitting – always a factor), porting to a new version of .NET might also help.
Possibly there are speed issues that are LAN-specific, however Duplicati does SMB through your OS.
You could help prove or disprove that by backing up to a local file to see if your speed gets any better.
It will probably never remove the benefit of deduplication, but will gradually be faster over time.
It’s your choice whether to go for fast transfer with storage of excess data, or reduce that work.
An evaluation based purely on transfer speed is incomplete. Ultimately use what fits your case.
Desired features are also situation dependent. I know of less featured programs that run faster.
EDIT:
The reason I doubt current design will go the way EaseUS appears to go is we left that in 2012.
Block-based storage engine describes why, and gives the story behind why this is Duplicati 2…
It’s a complex scheme though, and is still having bugs shaken out (slowly, as resources permit).
This almost has to be something like the worst average transfer speed of a non-initial backup, and is atypical of reports, even yours when you said 5Mb/s MAX. I’m assuming it’s actually in bytes not bits.
If one looks at the Duplicati status bar, the speed on its right is an average that only begins after first upload begins (that’s why it’s not even seen initially), and declines until the next due to the averaging.
I’m still curious if the topic title “over lan” has any relevance, but perhaps someone else can test local.
Again there’s a bit versus byte question, but I’m not sure how a week translates to either one of them. Large backups need different settings, especially blocksize which is tuned for a 100 GB backup, so if above sentence means 1.2 terabytes is less than 20% of the total source, you need to scale up some.
Sometimes blocksize raising will not be enough, as large file counts can also drive the block count up, because while blocksize controls the blocks of file data, a file also has metadata such as permissions.
Block counts beyond a million (or a few) slow down the SQL radically. I think this is also being studied, however tuning the SQL has been an ongoing process. There won’t likely be any miraculous speedup.
Bottom line in my opinion is that Duplicati doesn’t do large scale backups well, but if it can (after more improvements) do smaller ones reliably, fast enough, and with better features and UI, it has that place.
https://usage-reporter.duplicati.com/ shows 65 million backups per year, so it must be OK in some use. Certainly we can study yours further if you like, but if you find something better suited, use it of course.
EDIT:
Comparison category here has some comparisons.
Well, given that you say that you read it and according to you it proves that Duplicati will never backup faster than 5MB/s while what I actuaIly said is that I did a backup of 1.7 TB in 6 hours, which means 78 MB/s, I can’t understand what is ‘sense’ for you.