Just in the past week or so, I have 2+ systems that are hanging at the end of the backups to a WebDav server that I have local control of. One of them hangs on ‘deleting unwanted files’ where the log indicates a retry (usually unable to connect to remote server) and then a file rename attempt on one of the dlist files.
The other system shows waiting for upload to finish, again with 1 retry and ‘unable to connect to remote server’. No activity in taskmgr under duplicati server (all my jobs run as server pointing to DBs in /programdata). I’m posting the last few lines of the profiling log here:
Oct 28, 2020 8:42 AM: Backend event: Put - Retrying: duplicati-i14a3367663fd4aceac685cc4ac81e8e0.dindex.zip.aes (27.40 KB)
Oct 28, 2020 8:42 AM: Operation Put with file duplicati-i14a3367663fd4aceac685cc4ac81e8e0.dindex.zip.aes attempt 1 of 24 failed with message: Unable to connect to the remote server
Oct 28, 2020 8:41 AM: Backend event: Put - Started: duplicati-i14a3367663fd4aceac685cc4ac81e8e0.dindex.zip.aes (27.40 KB)
Oct 28, 2020 8:41 AM: ExecuteScalarInt64: INSERT INTO “Remotevolume” (“OperationID”, “Name”, “Type”, “State”, “Size”, “VerificationCount”, “DeleteGraceTime”) VALUES (204, “duplicati-i14a3367663fd4aceac685cc4ac81e8e0.dindex.zip.aes”, “Index”, “Temporary”, -1, 0, 0); SELECT last_insert_rowid(); took 0:00:00:00.000
Oct 28, 2020 8:41 AM: Starting - ExecuteScalarInt64: INSERT INTO “Remotevolume” (“OperationID”, “Name”, “Type”, “State”, “Size”, “VerificationCount”, “DeleteGraceTime”) VALUES (204, “duplicati-i14a3367663fd4aceac685cc4ac81e8e0.dindex.zip.aes”, “Index”, “Temporary”, -1, 0, 0); SELECT last_insert_rowid();
Oct 28, 2020 8:41 AM: CommitAfterUpload took 0:00:00:00.003
Oct 28, 2020 8:41 AM: Starting - CommitAfterUpload
Oct 28, 2020 8:41 AM: Backend event: Put - Completed: duplicati-b792e39a5d1fc43caaa6ea2d3a5157453.dblock.zip.aes (42.63 MB)
The log has been stuck on the ‘retrying dindex’ file for 15 minutes now and will stay here for several hours if I allow it.
If killed (not easy by the way), then 95% of the backup is restorable, but the db needs repairing.
I’ve tried list-broken and purge-broken commands which complete successfully, but the same symptoms appear on next backup run.
Disk space and/or corruption has been checked on local machines, no windows updates applied recently, and my webdav backend works flawlessly for all my other clients.
Any guidance on where to go next would be greatly appreciated!