Can anyone confirm what I’m seeing with Duplicati performance is at expected levels or am I doing something that is very negatively affecting the upload speed.
Installed current version Duplicati - 18.104.22.168_beta_2019-07-14
Windows server 2016 up to date patches.
128 GB RAM, 24 CPU cores Xeon Gold 6126. 10TB RAID 1 SAS 10K disk volume. NTFS 4K formatted. NTFS Deduplication enabled. 10 GbE
The upload files are Veeam backups 40 GB to 250 GB in size, dedupe friendly, not encrypted saved on above deduplicated Windows NTFS volume all in a single folder - total folder size about 1 TB across 10 files.
Using an S3 client - e.g. TNTdrive / Cyberduck I can upload these files to the Wasabi bucket at 7-10 MB/sec.This is the plain vbk file and no encryption is being added on or before upload.
Internet is 100 Mbps 1:1 fibre, and getting 10 MB/sec uploads is not unusual for me.
Using Duplicati I only get a maximum of about 400 KB/sec on the upload to Wasabi, no throttling is set.
I’m trying to understand why its 25X slower, since as it stands its making daily uploads like this infeasible and aside from this it looks to be quite promising a product.
Is what I’m seeing pretty much the performance I can expect or is this an aberration and I should be getting closer to line speed? I realise its internet but I have tested at various times of day for a couple days now, also on different ISPs and its pretty consistent what I’m getting.
The uploaded files when I look at the bucket are all 50 MB dblock AES files and a number of smaller dindex files.
Is my experience a result of having enabled encryption or is this being affected by (inline?) deduplication algorithms or something else. This is the initial upload.
I tried the same upload with default block size of 50 MB files, and no SSL or encryption and upload speed was only marginally better at 450-500 KB/sec.
Is this expected or am I missing anything here. Is an initial upload always slow and then speeds up after time when dedupe efficiencies kick in as daily backup deltas are added to the bucket? Thanks for any insights.