Moved source files - disk read speed extremely slow, checking for changes

I’ve reorganized the filestructure of what is being backed up, so upon backup it’s checking to see if these files exist in the backup already, presumably verifying that blocks with the same hashes already exist. Read speed is a steady 4 MB/s, with occasional stops. Reading the file myself is over 200 MB/s.

Current action: Backup_ProcessingFiles

Live log with it set to show up to profiling messages:

  • Jan 18, 2020 12:44 PM: Checking file for changes $path, new: True, timestamp changed: True, size changed: True, metadatachanged: True, 10/31/2018 8:06:59 AM vs 1/1/0001 12:00:00 AM

  • Jan 18, 2020 12:44 PM: New file $path

  • Jan 18, 2020 12:43 PM: Including path as no filters matched: $path

I’m running Duplicati - in the docker container on an unRAID server.

While the Live log does report multiple “checking for file changes” I’ve verified with “iotop and lsof -p -r 5” that this is the only file being read, and while 5 file handles for my RAM-backed tmp file are open for the duplicati, none are being written to. sqlite files are on a SSD.

Nothing else on the server is being read from or written to, and I’ve killed all other non-system processes as well.

If I just do a “dd if=$path of=/dev/null bs=4M status=progress conv=fsync” of that same file, it reads at over 200 MB/s.

I verified dd speeds both outside of docker and within docker via “docker exec -it duplicati bash”

The file itself is a compressed video file that has not changed, we’ll see if it also occurs on compressible data. top reports 85% idle, 10% iowait, mono-gen duplicati using 15% CPU. Plenty of free RAM, no swapping to disk, etc.

Thankfully this process doesn’t happen frequently, but I was planning on doing some regular full-remote-verifications of the backup files, presumably the same process would occur. It’s a rather large backup of family photos and videos.

Duplicati will need to re-hash any files that have been moved. How powerful of a processor is in your unRAID system? Are you restricting CPU usage on the docker container? Have you customized the --concurrency-block-hashers option or is it at the default?

It’s a an older but reasonably powerful CPU: i7-3770. CPU nor memory usage is constrained on the docker (no --cpu or --memory type parameters given). The system is pretty idle as mentioned: 85% idle, 10% iowait, mono-gen duplicati using 15% CPU

–concurrency-block-hashers has not been modified. I see the default should be 2 for that, should I be seeing two files being read? Nothing else shows up in lsof.

Seems like on other files (older .DNG RAW photos ~ 20 MB, losslessly compressed) it went much faster, at 100 MB/s. The files it seemed to struggle with from what I recall were large 10+ GB compressed 4K H264 videos and 50 GB google takeout .tgzs (which I’ve been meaning to extract)