Duplicati is extremely slow on even small mp4 files, Linux and Win10.
I installed Duplicati yesterday and I am using it with Wasabi Online Storage. I’ve noticed the speeds have dropped to around 100KB/s on Windows and 4KB/s under Linux when backing up mp4 files. The Windows file is large (2GiB), but the Linux file is small (138MiB). So far Linux has spent over 30min on the 138MiB file. I believe it is has actually been a few hours, but I wasn’t writing down the file name until 30min ago, so it could have been another file. Progress shows 2.3%. The Linux machine is dual core, 64-bit AMD E-350 Processor running at 1.2GHz. It’s not very powerful, but it’s more powerful than this. “top” shows 99% idle, 528MiB free RAM, 5.5GiB free swap.
What could be causing this much slow down?
“lsof” for the file shows “mono-sgen” has it open.
What Linux version is this? If you have a way to monitor network usage (and the rest of the system is idle enough) you might be able to see if the declining speed is real or the average dropping because it’s stuck.
Deeper in the networking, there are tools like netstat to show the ESTABLISHED connections (or lack) to Wasabi. I don’t know exactly what host if any Wasabi will show – again this works best on a quiet system.
Deeper yet would be something like tcpdump or Wireshark to get a packet trace of whatever is happening.
On the simpler side, for something that small you should be able to watch the server live log at Retry level. Profiling gives the details, but can get very noisy. You can also use –log-file and –log-file-log-level=profiling.
Perhaps you can try some of these on Windows if it’s working right (maybe still TBD), to watch its backup.
I’m running Mageia Linux 7. I used the RedHat RPM for the install. I don’t thin it’s the Internet connection or Wasabi. I’ve been doing multiple computers at the same time. An Ubuntu machine finished fairly quickly, as did my other Mageia 7 system. Average speeds when I was looking at them were jumping between 300K/s and 1.3M/s.
One of the things I did on the later systems was add the advanced option “compression-extension-file”. I didn’t modify the fie or the settings just added it. I thought this was enabled by default, but maybe it isn’t.
I tried the live log, but it didn’t show anything happening. I went through all of the options and the latest event was: “Aug 5, 2019 6:30 AM: CommitAddBlockToOutputFlush took 0:00:00:00.329”.
I’d say it was dead, but the system thinks it’s still running and the “mongo-sgen” process still has an active handle on the file.
I clicked stop in the WEb-GUI so that I could restart it with the log file enabled and I got a message asking me if I want to wait until the current file has been uploaded, so Duplicati thinks it’s running. I stopped the backup and all of the other ones that tried to start. I’ll restart with the log turned on and see what I get.
When I tried to restart the process it told me the database was locked. I tried to repair it but it told me there were no files. I tried delete and repair, but the same problem. I deleted the database and also the remote files (less than 8GiB out of 200GiB) and started over with the log on. I still haven’t set the file compression option yet. I’ll see what the log returns first.
It didn’t freeze on the file this time. It wasn’t quick, but it took care of it in a few minutes.
Live with Retry just shows a bunch of started and completed block and index zip.aes files.
I didn’t see anything interesting in the log file with the level set to profiling.
I’m leaving the logs in place for now. I’ll see if this backup finishes, still 258GiB to go.