I’m trying to backup my media library using Duplicati and I’m getting extremely slow average transfer speeds - around 2.85 MB/s. My library has almost 374,000 files totaling about 3.25TB and according to my calculations this would take almost 2 weeks. The files are almost all mpeg2, m2ts, cr2 (Canon RAW), and ARW (Sony RAW), and a few jpeg files. Duplicati is running with the default encryption and compression settings as well.
Duplicati is running on a QNAP NAS backing up to a new 8TB drive on a Raspberry Pi 2 (4 cores at 900Mhz) over SFTP. Everything is hard wired to the same switch in terms of networking. The load on the Pi is very low with a load average of 0.55/0.65/0.66, and the NAS also has a fairly low load of 1.66/1.79/1.85. Neither device is anywhere near their RAM limits either.
Is there anything I can do to speed up the initial backup? I don’t mind these speeds after the first backup since the whole Pi and HDD will be moved offsite and I won’t be able to sustain 3MB/s upload anyway.
You could pull the 8TB drive from the RPI and plug it into your media server. Then back up to it as local storage (provided that they can read and write the same file system type).
If it’s faster you can let it finish and then move the disk back to the RPI and change the backup job to point over SFTP again. It won’t matter how the backup accesses the backup files it just matters that they’re there.
Its in the KB/s range for about a minute and then spikes up to ~120MB/s (near the end of the video) for a second and then goes back down to the KB/s range. I’ve overlayed htop on the NAS and it looks like the bottleneck here is the CPU?
Would this mean that if I started the backup process on a computer with a faster CPU, the process should go by much faster? I’d have to mount my source data over Samba and I could probably connect the backup drive directly.
EDIT:
I’ve started the backup from a Windows computer with an i7 4790k in it and the backup is orders of magnitude faster. I’m where I was after over 12 hours on the NAS in a little over 20 minutes.
However, Duplicati recognized over 375,000 files when it ran on the NAS, while on the Windows computer its a little over 93,000 files. Both times the total size to backup was identical - 3.25TB but I have no idea why this large discrepancy in file count exists. I double checked and in each case, the backup source is identical.
That might be helpful. Otherwise the options might be either just waiting for it to finish or tweaking the --zip-compression-level to see if that helps performance at the cost of compression ratio. Note that the compression level has to be set before any data has been uploaded because there isn’t a built in way to change it later.
Just to clarify, you CAN change --zip-compression-level at any time without problem, however only compressed files created after the change will have the new level. Old files won’t be automatically re-compressed UNLESS they are part of a compact processes.
Good call being cautious! But as far as I know the only settings that can’t be changed are --blocksize--block-hash-algorithm and encryption passphrase. In all 3 cases my tests show that the CLI will throw an error and the GUI will show a message about not being able to change that setting.
Nope. You can change --block-size (aka “Upload volume size”) any time without problems however just as with --zip-compression-level the new value will only be applied to archives created going forward. Existing ones won’t be automatically re-compressed at the new size unless part of a compact process.
I believe there are some manual steps you can take to force a re-compress of everything, but I’ve never tried it myself. If you’re interested let me know and I’ll try to find the related posts.
I’m not sure what could be causing the discrepancy in file counts. Maybe it’s a bug in the file counter? The counter is a separate process from the actual backup so if it’s only in the counter it shouldn’t matter to the backup result. But I don’t know how you’d know until you finished the backup and moved back to the NAS:
I have mostly m2ts and arw files which are compressed but not found in that text file. I’ve edited the file to exclude those types as well. Hopefully that speeds things up a bit.
I’m assuming the encryption process is more CPU-bound than the compression process?
Default compression level is 9 and going down to 1 cut about 36% off his backup time in one of the tests. An other test using level 0 cut that down about 45%. And the test with turning off encryption cut about 7% (for a total of 52% with level 0 compression and no encryption)
Of course these things could also depend on memory and disk speeds, or even network speeds if you end up bottlenecked there.
I don’t know if that file is dynamically loaded or not - if adding those extensions works for you, please let us know so we can try and get them in the official release file.
I restarted Duplicati after making the changes - just to be sure.
I ran into the restoring from a different OS issue so I restarted the backup but the speeds seem to be significantly better. The Pi only has a 100Mbps port but I’m averaging around 45Mbps - still going to take 6 days to finish at this rate, but its half as long as the original estimate.
I’ll see if the transfer speed stays consistent over the next little while.
ARW is similar to Canon’s CR2. It should be an uncompressed image file but by most reports its actually a compressed format. I did (naively) a quick compression test with a few ARW files into a zip file and the resulting output was barely smaller than the original.
