CPU load from Duplicati: more than twice than from CrashPlan (may be up to 4)

#1

I am setting up Duplicati now as full replacement of CrashPlan. So, I can now compare the two on the same machine with roughly the same load (actually, the set of Duplicati is a lot smaller). Both backup programs have their sets up to date and they run with the same frequency (actually, CP with a higher frequency). Duplicati backs up up to B2.

CP takes about 50% of the CPU time of Duplicati. I already though CP was using much, but Duplicati much more. After 9 hours uptime, Duplicati has used more than two hours CPU, which is a lot. It uses 25% of this (admittedly, old and not very fast) Core2 Duo (which has SSD). ButCP only takes 1 hour and has more than twice the load, inlcuding backing up to a local disk (HD, slow) next to the remote service.

Probably the main reason is that CP uses macOS FSevents to limit file system access to that was has actually changed and Duplicati travels all of the files every time. And given that one set in Duplicati is 240k files and the other is probably around a quarter of that, that costs a lot of time.

Conclusion: at my scale, Duplicati really needs FSevents support. Either via a workaround (now) or in the code. Anybody up for a bounty? :slight_smile:

#2

The code has support for passing in a list of changed and deleted files, which would avoid the file scanning process.

But, so far, there is no monitor that keeps track of which files are changed (like FSEvents, USN, etc). The monitor needs to run with the server, and monitor the source paths. Whenever a path is noted as changed, it should be added to a list of changed files. On the next backup run, it should then simply feed this list via the --changed-files option.