That is really great work! And I am well aware that such setups take a long time to get up, and report on.
That is some comfort to me That means we need to tweak things to get there, and re-consider the default settings.
There is an issue here that says that the default hashing implementation is inefficient, and that we should use another:
It should be trivial to switch over:
It does nothing on Mono, but no ill effects:
Yes, it appears that compression is some of the problem. From what I read, CY uses LZ4, which is considered “fast”:
I can add LZ4 support to Zip:
But unfortunately, writing LZ4 streams inside a zip archive will make it incompatible with most Zip tools.
I will make a versions that uses SHA256Cng, to see if that takes the edge of the CPU usage.
I also have the concurrent_processing branch, which I have been working on for improving error handling. With that branch, it should be possible to stream data as a fast as the disk can provide it, without any of the “pauses” you see.
We can also consider storing the zip file in memory, instead of writing it to disk (
--write-volumes-to-disk ? ).