Server has wired conections to internet of 500Mbit, but always time i see an upload speed about 10-15MB/s. Aws cli work better, internet bandwidth has utilization about 90%. How can i make faster duplicati? I have files about 60GB and backup work really slow.
The server has no disk queues, enough free memory and the processor is not 100% loaded.
Duplicati does far more than a file copy. Also, processor loading is limited by how program uses cores.
max-concurrent-requests might be part of the faster file upload. You could try similar in Duplicati raising:
--asynchronous-concurrent-upload-limit (Integer): The number of concurrent
uploads allowed
When performing asynchronous uploads, the maximum number of concurrent
uploads allowed. Set to zero to disable the limit.
* default value: 4
Thanks for answer.
I found what was wrong. My S3 Cloud provider work only with s3-signatureversion 4. In main journal was a lot of error:
System.AggregateException: One or more errors occurred. —> System.AggregateException: The stream does not support concurrent IO read or write operations. —> System.NotSupportedException: The stream does not support concurrent IO read or write operations.
Backup worked but very slow and --asynchronous-concurrent-upload-limit don’t help me. Setting value s3-signatureversion to 2 doesn’t help either.
Only when I tried backup to another S3 cloud provider who work with s3-signatureversion 2 it’s work really faster.
could you be so kind to add this to the Duplicati documentation ? You have to create a Github account, but it’s not necessary to know git commands to do that, it’s possible to create a pull-request with the Web UI by clicking the Edit button on a manual page:
I think it’s not right what this work like that. Maybe it’s bug or old library in duplicati. New issue will be better. But i don’t know how fast this problem will be fixed.
I don’t use S3, but am confused by the problem, the resolution, and what is being proposed for manual.
which is the default (note actual name, and also, put these on Options screen 5 or they may disappear):
The names and defaults are (I believe – any C# dev want to look) read directly from AWS SDK for .NET:
What exactly? There are no provider names here yet, and I’m not sure we want to hardcode any in docs.
Code volunteers are so few that it may be time to name somewhere some big unfixable issues in queue.
Duplicati exists and improves only through code, docs, and support volunteers. Any who can – please do.
That topic ended with a possible provider-dependent solution that I’ve seen before.
Sounded odd, however it’s the wrong option (typo?), and the right one is 4 already.
Some are not fully S3 compatible. I’ve seen a few even documenting their subset.
But there are two intermixed issues here. You linked to a compatibility adjustment,
but you also switched providers, and found a faster one. I’m sure some are faster.
Mixes “working” and “working fast” needs. Yes, some providers can vary with both.
“Working” leads us to an unclear signature version situation that I’m asking about.
“Faster” might have more to do with other provider than with the signature version.
I’m still confused about your first provider. Did you try different signature versions?
Or if you’re happier with your second (which might just be faster), just stay there…
So should the GitHub issue be closed? Note that there is still an open question on signature version usage.
The issue detailed aws cli speed further. One possible reason it goes faster is due to multiple connections. How do I use the AWS CLI to perform a multipart upload of a file to Amazon S3?
I’m pretty sure Duplicati uploads a single file over one connection, whereas aws s3 cp might be using ten.
Duplicati can do different files concurrently over different connections. Unfortunately you found it didn’t help.
