I am trying to get the fastest speeds possible from a lan (Linux box connected via crossover cable to windows box). I can copy & paste my file and it takes 1min 15 sec . In duplicati the best I can get is 4min 53 sec.
While searching the forum it seems for best lan speed set configuration to:
–use-move-for-put=true
–disable-streaming-transfers
–zip-compression-level=0
–tempdir=/home/tmp/
At this point my goal is just to see what I can get for the fastest transfer. What am I missing?
Should also mention when I view my cpu process I bounce around 17-45%
Running Duplicati 2.0.3.3
I get a 10 second difference between 200MB or 10GB Upload volume size. I am backing up a single 10gb file - just for test purposes
Duplicati has build overheads for de-duplication and compression of data. Both of which are CPU intensive. First the file is hashed. Then it’s broken down into blocks and those blocks are hashed, then those blocks are compressed and stored in a dblock file. To top all that there are also database overheads in storing information about the files being backed up, blocks and hashes, dblocks, etc. In a nutshell you will never get near LAN speed with Duplicati.
Thanks Samw appreciate the explanation. I did read through the forums and found How can I speed up local backups? where kenkendk replied that using both the -use-move… & -disable-streaming… options would allow duplicati to bypass the buffer & be more like a copy & paste. Maybe I misunderstood that to mean the speeds would be more like explorer’s copy and paste & thought maybe I have a missed or wrong setting when I didn’t hit those copy paste speeds.
Does it still seem to be correct that the overhead would cost about 4min of time per 1min of copy & paste speed? In my test my cpu process never came close to maximizing & it’s being saved from an ssd to an ssd drive. The box is a duplicati dedicated i3 quadcore 8gb ram on linux lite if this helps
Is there another test or different way I should run it to increase the transfer time?
Are you sure the 4m53s is the transfer time? Duplicati tries to do multiple things in parallel, so the speed can be hard to measure. You can use something like --synchronous-upload to ensure that it does one thing at at time. It should then show as “active, pause, active, pause, …” in a bandwidth monitor.
The operations performed are generally:
Put blocks into zip volumes (compression overhead, temp-disk write)
Transfer the volume (read from temp-disk, write to destination)
For a non-LAN host, the transfer is usually the bottleneck. If you have the above flags set, and the destination appears as a filesystem entry (i.e. the destination url starts with file:// ), then the --disable-streaming-transfers --use-move-for-put options bypasses all processing in Duplicati (progress monitor, bandwith trottling, cancel checks etc), and simply issues the equivalent of mv tmpfile.zip.aes /remote/destname.zip.aes.
If nothing else is hammering around, you should get the same transfer speeds (ignoring the other stuff) as performing a mv operation on the commandline.
In addition to @kenkendk comments I would add the following. If space consumption and security are not issues and they shouldn’t be as it’s a LAN. You can also disable encryption altogether. Change the default block hash algorithm to SHA1 or MD5. Disable compression using the compression level=0 parameter assuming you are using the default zip compression. Also keep in mind that the beauty of Duplicati is subsequent backups which run a lot quicker than the first initial backup. Do your tests and do update us. Cheers.
I’m pretty sure it’s the transfer time because I have turned encryption off as well as compression. Which should just leave assembling the files into a zip folder (at least I have zip folders in my designated back up folder), I have also changed the files to be mostly jpg, avi mp3 files to backup. A simple copy paste is now 1 min even and with the changes in duplicati I am now 6:38sec
In Step 1 (General) I have encryption set to “No Encryption”, In step 5 (Options) under advanced this is what my config file is:
–tempdir=/home/backup/temp/
–use-move-for-put=true
–disable-streaming-transfers=true
–zip-compression-level=0
–thread-priority=highest
–synchrousronous-upload=true
Correct, security is not an issue & I have had the encryption turned off in step 1 in all the testing. I have left all the other settings the same and have now added the block hash algorithm to be MD5 and transfer time was the same at 6:38 just to be thorough I also tried SHA1 & it finished at the same time of 6:38
My config file looks like (No encryption in step 1)
–tempdir=/home/systems10/temp/
–use-move-for-put=true
–disable-streaming-transfers=true
–zip-compression-level=0
–thread-priority=highest
–synchrousronous-upload=true
–block-hash-algorithm=MD5
I do understand that subsequent backups will be quicker I was just using this as a gauge for how quick a restore would be which is an important factor
I don’t have any numbers to verify this, but I suspect a restore will be quite a bit faster than a backup for many reasons including that restores use much fewer (and very targeted database lookups) while backups have a lot of overhead doing very general database lookups to see if block hashes already exist.
Have you tried timing a restore of files you timed the initial backups of? If so, I’d be curious to know what sort of performance you get.
Also, have you timed doing the same backups to a local drive (instead of over the LAN / crossover cable) to see what performance is like with networking taken out of the picture? It might be a good way to help isolate where the bottleneck is occurring.
Well I’m trying to do a restore to the windows box I get thousands of errors when restoring. Even if I delete all files @ the orig location it does not restore. I can restore to the linux box but not the win10 even though “everyone” has full access to the “backup” folder. I can copy and paste into the win 10 backup folder no problem but using my share mnt point seems to be a problem
As for doing a speed test to the hard drive the restore was within a cpl minutes locally which is fast. I wasn’t going to try the backup locally as I was trying to get Duplicati to the same speed as a lan copy/paste which was that 1min 15sec. I assume the lan copy/paste was more realistic of true lan speed without the Duplicati overhead but with my settings I choose I thought they would have eliminated the overhead. I’ll post the errors on my lan restore
In case I haven’t already mentioned it, since you’re doing performance testing you might want to set --no-local-blocks=true otherwise Duplicati will attempt to use unchanged blocks from the already local source files as part of the restore.
--no-local-blocks
Duplicati will attempt to use data from source files to minimize the amount of downloaded data. Use this option to skip this optimization and only use remote data.
Default value: “false”