[resolved] Backup fails, but only on schedule

Support
#1

i have a backup from ubuntu desktop to an offsite s3 server running minio. it runs on a nightly schedule around 3 AM. it had generally been consistently working without issue for quite some time. about a week ago it started failing during scheduled backups. the next morning i ran a manual “run now” backup and it worked without any issues. the next night the scheduled backup failed. for the last week every scheduled backup has failed and every manually run backup has succeeded. the error is consistent, but i can’t recreate it except on the schedule.

image
image1906×864 43.8 KB

Failed: Expected hash not equal to calculated hash
Details: Amazon.Runtime.AmazonClientException: Expected hash not equal to calculated hash
at Duplicati.Library.Main.AsyncDownloader+AsyncDownloaderEnumerator+AsyncDownloadedFile.get_TempFile () [0x00008] in <8f1de655bd1240739a78684d845cecc8>:0
at Duplicati.Library.Main.Operation.CompactHandler.DoCompact (Duplicati.Library.Main.Database.LocalDeleteDatabase db, System.Boolean hasVerifiedBackend, System.Data.IDbTransaction& transaction, Duplicati.Library.Main.BackendManager sharedBackend) [0x00264] in <8f1de655bd1240739a78684d845cecc8>:0
at Duplicati.Library.Main.Operation.DeleteHandler.DoRun (Duplicati.Library.Main.Database.LocalDeleteDatabase db, System.Data.IDbTransaction& transaction, System.Boolean hasVerifiedBacked, System.Boolean forceCompact, Duplicati.Library.Main.BackendManager sharedManager) [0x00397] in <8f1de655bd1240739a78684d845cecc8>:0
at Duplicati.Library.Main.Operation.BackupHandler.CompactIfRequired (Duplicati.Library.Main.BackendManager backend, System.Int64 lastVolumeSize) [0x000a5] in <8f1de655bd1240739a78684d845cecc8>:0
at Duplicati.Library.Main.Operation.BackupHandler.RunAsync (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x01033] in <8f1de655bd1240739a78684d845cecc8>:0
at CoCoL.ChannelExtensions.WaitForTaskOrThrow (System.Threading.Tasks.Task task) [0x00050] in <9a758ff4db6c48d6b3d4d0e5c2adf6d1>:0
at Duplicati.Library.Main.Operation.BackupHandler.Run (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x00009] in <8f1de655bd1240739a78684d845cecc8>:0
at Duplicati.Library.Main.Controller+<>c__DisplayClass14_0.<Backup>b__0 (Duplicati.Library.Main.BackupResults result) [0x0004b] in <8f1de655bd1240739a78684d845cecc8>:0
at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String[]& paths, Duplicati.Library.Utility.IFilter& filter, System.Action`1[T] method) [0x0011c] in <8f1de655bd1240739a78684d845cecc8>:0

Log data:
2020-12-30 03:18:38 -05 - [Error-Duplicati.Library.Main.Operation.BackupHandler-FatalError]: Fatal error
Amazon.Runtime.AmazonClientException: Expected hash not equal to calculated hash
at Duplicati.Library.Main.AsyncDownloader+AsyncDownloaderEnumerator+AsyncDownloadedFile.get_TempFile () [0x00008] in <8f1de655bd1240739a78684d845cecc8>:0
at Duplicati.Library.Main.Operation.CompactHandler.DoCompact (Duplicati.Library.Main.Database.LocalDeleteDatabase db, System.Boolean hasVerifiedBackend, System.Data.IDbTransaction& transaction, Duplicati.Library.Main.BackendManager sharedBackend) [0x00264] in <8f1de655bd1240739a78684d845cecc8>:0
at Duplicati.Library.Main.Operation.DeleteHandler.DoRun (Duplicati.Library.Main.Database.LocalDeleteDatabase db, System.Data.IDbTransaction& transaction, System.Boolean hasVerifiedBacked, System.Boolean forceCompact, Duplicati.Library.Main.BackendManager sharedManager) [0x00397] in <8f1de655bd1240739a78684d845cecc8>:0
at Duplicati.Library.Main.Operation.BackupHandler.CompactIfRequired (Duplicati.Library.Main.BackendManager backend, System.Int64 lastVolumeSize) [0x000a5] in <8f1de655bd1240739a78684d845cecc8>:0
at Duplicati.Library.Main.Operation.BackupHandler.RunAsync (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x00d9f] in <8f1de655bd1240739a78684d845cecc8>:0

i checked the minio server, it was not restarted or otherwise down during the backup. i’ve tried restarting both client and server.

any ideas?

thanks.

#2

i’m not totally sure what changed, but i was able to force the error locally after a few tries. i think i may have changed the s3 sdk to/from aws or minio. while running i was able to log/profile and it provided a specific file it was choking on. i deleted that, purged and now it is back to running manually again. we’ll see what happens tonight…

#3

that seemed to do the trick. weird that i hadn’t seen that particular error before.

1 Like