I have just installed 2.0.4.18_canary_2019-05-12 and the backup speed (to network folder) is 1-2MB/s. I am trying to backup ~3TB of data, so it will take more than a week. The system is Windows2008R2, CPU is i5-2300. When i copy files from src to dst the speed is ~80MB/s, cpu usage by duplicati is constantly jumping from 0 to ~30, but is not going higher so it doesnt seems to be a cpu bottleneck either.
I am using duplicati on other machines and never seen anything like this.
P.S. I looked at perfmon, actual transfer over network happens really fast, so its definitely not a transfer to destination speed issue. What bothers me is that it reads many (up to ~25) files from src at the same time, i suspect thats what kills read speeds…
Does that mean this is a new install (not an update)? Canary sometimes has bugs, so testing an older version (2.0.4.15 or a beta) might be interesting. The new parallel upload code can also be adjusted by setting --asynchronous-concurrent-upload-limit in Advanced options. Default is 4 and should run faster.
I’m not sure what effect new uploader has on disk reads. I guess you’re fearing mechanical drive random access reads run slower than reading a file (or a smaller number) at a time. Does Task Manager indicate that your drive is constantly busy? I’d have expected an initial backup to not have to work hard to find data to fill the network (are you saying uploading is bursty?) whereas later backups might have to scan harder.
You can use a llive log or log-file at Information level or higher to watch Duplicati put dblock files to remote to see if it looks bursty there. Especially with parallel uploads, I’d have expected several uploading at once.
The uploads almost doesnt register in perfmon, it happens very fast - the block size is set to 200MB and it writes this 200MB to destination in less than 5 seconds. Network graph looks flat for minutes, than burst at full speed than again flat for minutes. On the disk side it reads from 10-25 files (with a total speed of 3-4MB/s) and writes to a ~5 files in temp at the same speed. This disk read speed is consistent, so its not burst read than wait, according to perfmon it actually reads from multiple files very slow. Temp is on a different physical disk from source of backup so shouldnt affect it all.
I cant access live log, when i click log in interface i get “database locked” error.
P.S. Scratch the “many files” part, its some artefact of perfmon. When the backup got to a large files, its reading from only one file at the speed of ~4MB/s. I can copy that same file in parallel from scr to a disk where temp files are written at speed of 160MB/s
P.P.S I tried last non-canary version (Duplicati - 2.0.4.5_beta_2018-11-28), read speed is slightly better (in the 10-15MB/s range). But its still unacceptably slow for 3TB backup…
I think it has been since April of 2021 that I began trying to get our 12+ TB of data uploaded to Backblaze. I will spare you all of the details, but it has been brutal!
I finally got it going and was averaging around 105 GB per day uploaded. (I know…right!) All of our data finally finished uploading last week and I started the automatic backups. All was going fine until about 3 days ago. The largest bucket, around 6.4 TB got into some kind of a thing where the speed dropped to 12KB (yes, 12 KBps) and there were over 5TB to go. I tried restarting the server multiple times, I restarted the Linux server several times, etc.
On a whim, I just started updating hardware on the server where the Duplicati GUI is installed. There are 4 network drivers. I updated them one at a time. When I got to the 4th, (I’ll be dipped!) the network connection dropped for about 15 seconds and came back online. The file size dropped from 5TB to about 90 MB!!! The backup went through its routine and completed. Duplicati reported 400someoddish errors, but the next bucket began backing up per the schedule!
For some reason, it’s not showing the rate per second, but this bucked was about 1.6TB. It’s about half done and it has only been about 45 minutes or so.
This may have just been a fluke, but check your NIC drivers if you’re having extreme speed issues.