I’m getting terrible performance on both a Windows 10 machine, and Windows Server 2016 NAS. Backing up anything is painfully slow. The bottleneck seems to be CPU, with one core being constantly hit hard, but with read/writes to disk and over the network being minimal. It’s taking on average a minute…

There currently isn’t a “here are the slow parts” profiling function, but the Live Log does have a profiling mode that shows pretty much everything being done and how long it takes, but it’s not pretty. Some discussion has happened around adding metrics, but so far development time has been focused…

This is the initial backup that’s really slow unfortunately. I’ve left it overnight and it’s managed 500GB in 8 hours. These are video files it’s backing up, so there’s no reason for it to be so slow. Copying the files manually would take about 80 minutes at the most, so duplicati is at least 7x slo…

[image] Wardrop: it seems the call to mscorlib.dll!System.Security.Cryptography.SHA256Managed.SHATransform is likely responsible It sounds like the hashing of the blocks is the bottleneck you’re running into. I should have asked earlier, but what version are you running? I believe the later…

This is a CPU bottleneck, not network or I/O. It maxes out a single core, and that’s what caps the speed.

[image] Wardrop: This is a CPU bottleneck, not network or I/O. That would line up with inefficient block hashing. Were you able to determine what Duplicati version you are running? If it’s the latest canary version and still slow then the updated hashing code may but be giving as much of a …

I’m running 2.0.2.1_beta_2017-08-01, which seems like it’s a few months old now. I downloaded it off the website only in the last week. Going by profiling sample below, it’s not necessarily the hashing function, which itself is only consuming 30% of the CPU cycles, but it seems it’s rather what hap…

Canary really wasn’t much different. Same speeds, maybe a couple of MB/s faster. Here’s the profiled call tree for it… [Duplicati Profile (Canary)]

Nice profiling, thanks for sharing that! I’m not an expert but it looks like it’s the block processing (which is one area where Duplicati varies widely from a simple file copy) causing the issue for you. It’s POSSIBLE the still-in-testing feature that will (where appropriate) allow moving block ha…

When I really drill into it, it works out to be 33% of CPU time spent on zip compression, and 62% spent on calculating hashes. Only ~1% of the CPU time is spent doing anything else. I think to improve the situation, firstly, compression needs to be made optional, as that 33% spent on compression wa…

My bad, I didn’t notice you can set the compression method to “none”. Doing that boosted throughput by pretty much exactly 33% as the profiling predicted. Changing the hash algorithm from SHA256 to MD5 gave a ~15%. improvement. Combining them nearly got us to 50% overall improvement. Still not quick…

Duplicati

Performance is really slow with high CPU usage!

Support

show post in topic

Home
Categories
Guidelines
Terms of Service
Privacy Policy

Powered by Discourse, best viewed with JavaScript enabled