Is this normal performance, taking a day for less than 400GB to a local external SATA drive? I started slightly before noon and 9 hours later, only 50% done.
With the old Crashplan, the same dataset could be done from scratch in less than two hours using the same drive after having been formatted.
A lot of factors can affect performance so it’s hard to say what’s “normal”. That being said, comparing performance of Duplicati to CrashPlan is best due done after tweaking some of Duplicati’s default settings to more closely match how we GUESS CrashPlan works.
Of course speed isn’t everything, so be sure to consider disk space used (local and destination) as well as other resources (Duplicati likely not uses less memory).
And be sure you’re testing similar cases. For example, is CrashPlan sleeping (or better yet the service shut down) when Duplicati runs?
All that sounds like I’m making excuses for Duplicati, but I’m not. CrashPlan may easily be faster than Duplicati during the actual backup. For example, Duplicati scans the source folder when the backup stars while CrashPlan seems to constantly be scanning whether a backup is running or not, so when one does start all has less to do compared to Duplicati.
Personally, on my machines the lower resources used by Duplicati when NOT backing up are worth a potentially slower backup.
Please note that depending on what Duplicati version you are running you may not have the latest performance engagements. On top of that there are some more that haven’t made it to a release yet.
Perhaps somebody who has done some actual benchmarks with Duplicati can chime in…
First, it’s still running as I type this reply, 30GB left to go 
At this point, I need to switch from Crashplan so any help finding a replacement is welcomed. Doesn’t feel like an excuse either
I wanted to add some details:
Edit
@Dan, what OS are you running on?
200GB in 9 hours is incredibly slow, that’s about 53Mbps or 6.5MB/s. Can’t think of any reason why it’d be going that slow. What kind of CPU do you have?
The closest I had to test with was setting up a local backup to an internal 5900RPM SATA disk. 200GB (~18K files) completed an initial backup in a little under 3 hours with mostly default settings. While it was backing up it was maxing out a single core, and using between 100-150MB of RAM.
Of the three non-default things I’m doing, only the first one is likely to have much impact on performance:
I don’t think Duplicati is the fastest (very interested in performance after multi-threading gets added), but it is adequately quick and I haven’t experienced it being infuriatingly slow either. And I second @JonMikelV, I’d trade a good bit of speed for the refreshingly low memory utilization compared to the gigabytes of RAM CrashPlan would chew up.
Same here. 
I’m also curious why some people seem to be getting such slow speeds. I know there are lots of external factors that can reduce speed (good catch on AV) but it would be great if we could identify some of the more common ones.
My initial 200GiB backup to a NAS also took 9h, and the second run (with 6GiB new data) took 22 Minutes. There is a strong disk activity in the temp folder and 50% CPU load on my i5-6200.
Info on my system and OS:
How do I run this as a service? I wasn’t able to find it in my Services (well nothing with Duplicati in it).
And lastly, what is VSS?
This How-To explains how to migrate Duplicati from a user-based to a service-based setup:
This video shows how to register the service after a fresh install:
VSS is a technology that enables backup software to make a copy of files that are in use by another process. More information can be found here:
What volume size are you using, out of curiosity? I’ve talked about this elsewhere – for a local drive backup, I found that the default 50MB volume size is way too small, as Duplicati ends up spending most of its time preparing and compressing the data blocks, instead of spending its time utilizing the upload / local drive bandwidth.
For mine I settled on 2GB volume sizes (NOTE: again this is for LOCAL backups only, you should not do this for B2), because it ends up with quite a bit less clutter and less file processing overhead as far as I can tell. I’m not sure what official benchmarks would show though, i’ll have to try it soon. I’d also suggest trying out 500MB and/or 1GB volume sizes if 2GB sounds too big - even the smallest of these is 10x fewer dblock files than the default size.
Yeah, used the default which is 50MB. I can retry using a bigger volume size and with the suggestions you provided earlier.
Dan, what version of Duplicati you’re running. @kenkendk recently switched Duplicati to use a faster hashing mechanism in the latest Canary builds.
I’m running on 2.0.2.1_beta_2017-08-01
I will attempt to test post-service install and bigger blocks. Will report back.
FWIW, I did a test with a bigger block size (500MB) and was seeing high writes to disk, sometimes it peaks in the 100MB/s. However, it fluctuates quite a bit, coming back down below 20MB/s. Previously, I think it was pretty constant at 2xMB/s.
After a short while, I nuked my external HDD, started VSS (it was set as manual startup) and installed Duplicati as a service. I seem to be getting the same performance as I had by simply increasing the block size.
@drakar2007 @sanderson The transfer is still going… 7+ hours.
Check out this Task Manager screenshot:
https://i.imgur.com/1CLNK4K.png
Notice C: utilization constantly at the top, and I: my backup drive, being written sporadically.
Edit
Got this shot of the Task Manager just now:
C: 100% I: barely anything
Lots of moving parts to keep track off …
There is a performance measurement here:
It suggests setting --zip-compression-level=1 and --no-encryption to get speedups.
The disk usage could be related to the SQLite transactions being written, hammering in with the temporary files as well.
@Dan, if you have a second local drive you could point your temp folder there and see if the high disk usage follows it.
It may not be a “fix” but it could help narrow down a specific cause of you slowness.
I tried the --zip-compression-level=1 and I did notice more frequent writes to my external HDD. However, we’re still talking 10+ hours and it still had 100GB+ to go. I want encryption in there though.
Unfortunately, I don’t have a spare drive. The other partition (ie. F) contains the folder that I’m going to be backing up (movies, pictures). Not sure if it’s wise to put the temp file on that drive.
Theoretically if you have “standard Windows exclusions” set on then Duplicati will ignore the files even if they’re in a folder being backed up.
As for the wise-ness of it, I don’t know you’d want to leave it there.
If you have time to test it, we are considering using in-memory storage when building the volumes.
You can set up a ram disk, and then set it to be the --tempfolder, which should reveal if the disk is really the issue:
If this shows significant speedup, I will prioritize the changes to get “in-memory” volume support.
