Slow Backups & Completion Error


#1

Hello,

I just installed Duplicati on Friday to 5 servers backing up to OD4B. Few issues. Running 2.0.3.4_canary_2018-04-02 on both Windows & Ubuntu (headless setup)

  1. Duplicati takes an average of 1 hour to backup on each server with very little changes. I have a 400GB still on it’s initial backup going on 14 hours. 32gb ram, gig ethernet. Borg can back up within just a few minutes, with rclone taking another few minutes. Is this normal? I do notice Duplicati has much better compression, but should the backup really take this long? My smaller servers of 2-3GB even take an hour. Is there some setting I have causing this? In comparison, Borg takes like 45 seconds to complete on the smaller servers. The backups are of Ubuntu and Windows Server 2016 VM’s.

  2. Receiving error of the following after backup finishes. Is this a bug or am I missing something?
    Errors: [
    2018-04-09 01:37:46 -04 - [Error-Duplicati.Library.Main.Operation.TestHandler-FailedToProcessFile]: Failed to process file duplicati-20180408T192200Z.dlist.zip.aes,
    2018-04-09 01:53:35 -04 - [Error-Duplicati.Library.Main.Operation.TestHandler-FailedToProcessFile]: Failed to process file duplicati-i8d36a5f5f18a49ad8dbb26acedfc3374.dindex.zip.aes,
    2018-04-09 02:09:25 -04 - [Error-Duplicati.Library.Main.Operation.TestHandler-FailedToProcessFile]: Failed to process file duplicati-b2dad2dfb795f452288c946c884135de6.dblock.zip.aes

Love the software, hope someone can provide some tips. Thank you!


#2

Hello,
I’ve the same error on point 2 on my fresh installation on Windows Server 2008 non-R2,
Some tips?


#3

Hello @Newdup and @acnb, welcome to the forum!

Performance expectations are a tricky thing to manage due to all the things that can very from system to system and configuration to configuration. Are you backing up the raw VM image files or is Duplicati running inside the VM?

For the “Failed to process file” error I suspect this is happening during the verification step that checks a set (dlist, dindex, and dblock) of destination files at the beginning of each backup run. My guess is the backups are still running, just throwing this error at you when they finish.

To confirm this, can you both try adding the --no-backend-verification=true advanced parameter value to your backup job?

--no-backend-verification
If this flag is set, the local database is not compared to the remote filelist on startup. The intended usage for this option is to work correctly in cases where the filelisting is broken or unavailable.
Default value: “false”

Note that this is a test and NOT a solution as it basically TURNS OFF some of the tests that Duplicati does to make sure you backups are viable. The actual cause of the issue could be something as simple as the temp folder to which Duplicati is trying to save the files is full, maybe Duplicati doesn’t have file read rights to the destination, etc.


#4

Hi @JonMikeIV

Same problem here and I just tried --no-backend-verification. Backup process ends without any errors.

Your suggestions about temp folder (enough space here and there) or read rights (write/read rights on both side) doesn’t help me to figure out what’s the problem. Any ideas?


#5

So with --no-backend-verification enabled the job runs without problem?

Then we should see if there’s more detail about the “Failed to process file duplicati-”. What I suspect is happening is that during the verification step when Duplicati downloads a set of files (dblock, dlist, and dindex) to be tested something like the following is happening:

  • the download is failing (so nothing to test)
  • the download results in a corrupted file (so test fails, but this should result in a different error)
  • the download works but before the test can be conducted the files are deleted (perhaps disk cleanup or antivirus is removing the files)