Duplicati reporting errors - new installation

I have been testing Duplicati as my backup solution. I have a new Synology DS220j NAS and have set up one backup task that is reporting errors which I don’t understand what it is failing and more important what can I do about it.

There are 12 warnings reported, here is a typical one (the rest are the same but for a different file).

  • 2021-09-01 13:00:01 +01 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingRemoteHash]: remote file duplicati-b93a91a657a2047208b25716133b79538.dblock.zip.aes is listed as Uploaded with size 48234496 but should be 52330637, please verify the sha256 hash “iC5Pw8ClIpPuQI4T7WmInS2MfcGGmoje0BznevXiq3g=”

What could have caused this error? It might be configuration error on my new NAS. All 12 errors are similar with different encrypted files.

How can I correct it?

I have previously reported irrecoverable errors. My old NAS died and I have replaced it and it is now the new installation that has failed. It was assumed that the failing NAS created the errors

Here s the full log
{
“DeletedFiles”: 0,
“DeletedFolders”: 0,
“ModifiedFiles”: 0,
“ExaminedFiles”: 33249,
“OpenedFiles”: 0,
“AddedFiles”: 0,
“SizeOfModifiedFiles”: 0,
“SizeOfAddedFiles”: 0,
“SizeOfExaminedFiles”: 11235973782,
“SizeOfOpenedFiles”: 0,
“NotProcessedFiles”: 0,
“AddedFolders”: 0,
“TooLargeFiles”: 0,
“FilesWithError”: 0,
“ModifiedFolders”: 0,
“ModifiedSymlinks”: 0,
“AddedSymlinks”: 0,
“DeletedSymlinks”: 0,
“PartialBackup”: false,
“Dryrun”: false,
“MainOperation”: “Backup”,
“CompactResults”: {
“DeletedFileCount”: 2,
“DownloadedFileCount”: 0,
“UploadedFileCount”: 0,
“DeletedFileSize”: 2026,
“DownloadedFileSize”: 0,
“UploadedFileSize”: 0,
“Dryrun”: false,
“VacuumResults”: null,
“MainOperation”: “Compact”,
“ParsedResult”: “Success”,
“Version”: “2.0.6.3 (2.0.6.3_beta_2021-06-17)”,
“EndTime”: “2021-09-01T12:00:16.981458Z”,
“BeginTime”: “2021-09-01T12:00:16.6079004Z”,
“Duration”: “00:00:00.3735576”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null,
“BackendStatistics”: {
“RemoteCalls”: 9,
“BytesUploaded”: 2026,
“BytesDownloaded”: 55019271,
“FilesUploaded”: 2,
“FilesDownloaded”: 3,
“FilesDeleted”: 2,
“FoldersCreated”: 0,
“RetryAttempts”: 0,
“UnknownFileSize”: 0,
“UnknownFileCount”: 1,
“KnownFileCount”: 290,
“KnownFileSize”: 7491841132,
“LastBackupDate”: “2021-08-31T22:59:12+01:00”,
“BackupListCount”: 2,
“TotalQuotaSpace”: 3931605622784,
“FreeQuotaSpace”: 3828505280512,
“AssignedQuotaSpace”: -1,
“ReportedQuotaError”: false,
“ReportedQuotaWarning”: false,
“MainOperation”: “Backup”,
“ParsedResult”: “Success”,
“Version”: “2.0.6.3 (2.0.6.3_beta_2021-06-17)”,
“EndTime”: “0001-01-01T00:00:00”,
“BeginTime”: “2021-09-01T12:00:00.0508877Z”,
“Duration”: “00:00:00”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null
}
},
“VacuumResults”: null,
“DeleteResults”: {
“DeletedSetsActualLength”: 0,
“DeletedSets”: ,
“Dryrun”: false,
“MainOperation”: “Delete”,
“CompactResults”: {
“DeletedFileCount”: 2,
“DownloadedFileCount”: 0,
“UploadedFileCount”: 0,
“DeletedFileSize”: 2026,
“DownloadedFileSize”: 0,
“UploadedFileSize”: 0,
“Dryrun”: false,
“VacuumResults”: null,
“MainOperation”: “Compact”,
“ParsedResult”: “Success”,
“Version”: “2.0.6.3 (2.0.6.3_beta_2021-06-17)”,
“EndTime”: “2021-09-01T12:00:16.981458Z”,
“BeginTime”: “2021-09-01T12:00:16.6079004Z”,
“Duration”: “00:00:00.3735576”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null,
“BackendStatistics”: {
“RemoteCalls”: 9,
“BytesUploaded”: 2026,
“BytesDownloaded”: 55019271,
“FilesUploaded”: 2,
“FilesDownloaded”: 3,
“FilesDeleted”: 2,
“FoldersCreated”: 0,
“RetryAttempts”: 0,
“UnknownFileSize”: 0,
“UnknownFileCount”: 1,
“KnownFileCount”: 290,
“KnownFileSize”: 7491841132,
“LastBackupDate”: “2021-08-31T22:59:12+01:00”,
“BackupListCount”: 2,
“TotalQuotaSpace”: 3931605622784,
“FreeQuotaSpace”: 3828505280512,
“AssignedQuotaSpace”: -1,
“ReportedQuotaError”: false,
“ReportedQuotaWarning”: false,
“MainOperation”: “Backup”,
“ParsedResult”: “Success”,
“Version”: “2.0.6.3 (2.0.6.3_beta_2021-06-17)”,
“EndTime”: “0001-01-01T00:00:00”,
“BeginTime”: “2021-09-01T12:00:00.0508877Z”,
“Duration”: “00:00:00”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null
}
},
“ParsedResult”: “Success”,
“Version”: “2.0.6.3 (2.0.6.3_beta_2021-06-17)”,
“EndTime”: “2021-09-01T12:00:16.981458Z”,
“BeginTime”: “2021-09-01T12:00:15.9609795Z”,
“Duration”: “00:00:01.0204785”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null,
“BackendStatistics”: {
“RemoteCalls”: 9,
“BytesUploaded”: 2026,
“BytesDownloaded”: 55019271,
“FilesUploaded”: 2,
“FilesDownloaded”: 3,
“FilesDeleted”: 2,
“FoldersCreated”: 0,
“RetryAttempts”: 0,
“UnknownFileSize”: 0,
“UnknownFileCount”: 1,
“KnownFileCount”: 290,
“KnownFileSize”: 7491841132,
“LastBackupDate”: “2021-08-31T22:59:12+01:00”,
“BackupListCount”: 2,
“TotalQuotaSpace”: 3931605622784,
“FreeQuotaSpace”: 3828505280512,
“AssignedQuotaSpace”: -1,
“ReportedQuotaError”: false,
“ReportedQuotaWarning”: false,
“MainOperation”: “Backup”,
“ParsedResult”: “Success”,
“Version”: “2.0.6.3 (2.0.6.3_beta_2021-06-17)”,
“EndTime”: “0001-01-01T00:00:00”,
“BeginTime”: “2021-09-01T12:00:00.0508877Z”,
“Duration”: “00:00:00”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null
}
},
“RepairResults”: null,
“TestResults”: {
“MainOperation”: “Test”,
“VerificationsActualLength”: 3,
“Verifications”: [
{
“Key”: “duplicati-20210831T205719Z.dlist.zip.aes”,
“Value”:
},
{
“Key”: “duplicati-ib6736e4adc9f4d5e8b8d7cd8b9710aa6.dindex.zip.aes”,
“Value”:
},
{
“Key”: “duplicati-b945257e49ec249afa12d9a62a8cf6b4e.dblock.zip.aes”,
“Value”:
}
],
“ParsedResult”: “Success”,
“Version”: “2.0.6.3 (2.0.6.3_beta_2021-06-17)”,
“EndTime”: “2021-09-01T12:00:20.6646578Z”,
“BeginTime”: “2021-09-01T12:00:18.9888641Z”,
“Duration”: “00:00:01.6757937”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null,
“BackendStatistics”: {
“RemoteCalls”: 9,
“BytesUploaded”: 2026,
“BytesDownloaded”: 55019271,
“FilesUploaded”: 2,
“FilesDownloaded”: 3,
“FilesDeleted”: 2,
“FoldersCreated”: 0,
“RetryAttempts”: 0,
“UnknownFileSize”: 0,
“UnknownFileCount”: 1,
“KnownFileCount”: 290,
“KnownFileSize”: 7491841132,
“LastBackupDate”: “2021-08-31T22:59:12+01:00”,
“BackupListCount”: 2,
“TotalQuotaSpace”: 3931605622784,
“FreeQuotaSpace”: 3828505280512,
“AssignedQuotaSpace”: -1,
“ReportedQuotaError”: false,
“ReportedQuotaWarning”: false,
“MainOperation”: “Backup”,
“ParsedResult”: “Success”,
“Version”: “2.0.6.3 (2.0.6.3_beta_2021-06-17)”,
“EndTime”: “0001-01-01T00:00:00”,
“BeginTime”: “2021-09-01T12:00:00.0508877Z”,
“Duration”: “00:00:00”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null
}
},
“ParsedResult”: “Warning”,
“Version”: “2.0.6.3 (2.0.6.3_beta_2021-06-17)”,
“EndTime”: “2021-09-01T12:00:20.6736213Z”,
“BeginTime”: “2021-09-01T12:00:00.0508877Z”,
“Duration”: “00:00:20.6227336”,
“MessagesActualLength”: 30,
“WarningsActualLength”: 12,
“ErrorsActualLength”: 0,
“Messages”: [
“2021-09-01 13:00:00 +01 - [Information-GetGpgProgramPath-gpg]: C:\ProgramData\Duplicati\updates\2.0.6.3\win-tools\gpg.exe”,
“2021-09-01 13:00:00 +01 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started”,
“2021-09-01 13:00:01 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: ()”,
“2021-09-01 13:00:01 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: (291 bytes)”,
“2021-09-01 13:00:15 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-bd2bcbd99670e4b0992b2bb196d58176b.dblock.zip.aes (1.08 KB)”,
“2021-09-01 13:00:15 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-bd2bcbd99670e4b0992b2bb196d58176b.dblock.zip.aes (1.08 KB)”,
“2021-09-01 13:00:15 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-ida05bc11d23a4429928869bfe9e1ca11.dindex.zip.aes (925 bytes)”,
“2021-09-01 13:00:15 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-ida05bc11d23a4429928869bfe9e1ca11.dindex.zip.aes (925 bytes)”,
“2021-09-01 13:00:15 +01 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-StartCheck]: Start checking if backups can be removed”,
“2021-09-01 13:00:15 +01 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-FramesAndIntervals]: Time frames and intervals pairs: 7.00:00:00 / 1.00:00:00, 28.00:00:00 / 7.00:00:00, 365.00:00:00 / 31.00:00:00”,
“2021-09-01 13:00:15 +01 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-BackupList]: Backups to consider: 31/08/2021 21:57:19”,
"2021-09-01 13:00:15 +01 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-BackupsToDelete]: Backups outside of all time frames and thus getting deleted: ",
"2021-09-01 13:00:15 +01 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-AllBackupsToDelete]: All backups to delete: ",
“2021-09-01 13:00:16 +01 - [Information-Duplicati.Library.Main.Operation.DeleteHandler-DeleteResults]: No remote filesets were deleted”,
“2021-09-01 13:00:16 +01 - [Information-Duplicati.Library.Main.Database.LocalDeleteDatabase-CompactReason]: Compacting because there are 1 fully deletable volume(s)”,
“2021-09-01 13:00:16 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Started: duplicati-bd2bcbd99670e4b0992b2bb196d58176b.dblock.zip.aes (1.08 KB)”,
“2021-09-01 13:00:16 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Completed: duplicati-bd2bcbd99670e4b0992b2bb196d58176b.dblock.zip.aes (1.08 KB)”,
“2021-09-01 13:00:16 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Started: duplicati-ida05bc11d23a4429928869bfe9e1ca11.dindex.zip.aes (925 bytes)”,
“2021-09-01 13:00:16 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Completed: duplicati-ida05bc11d23a4429928869bfe9e1ca11.dindex.zip.aes (925 bytes)”,
“2021-09-01 13:00:16 +01 - [Information-Duplicati.Library.Main.Operation.CompactHandler-CompactResults]: Deleted 2 files, which reduced storage by 1.98 KB”
],
“Warnings”: [
“2021-09-01 13:00:01 +01 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingRemoteHash]: remote file duplicati-b93a91a657a2047208b25716133b79538.dblock.zip.aes is listed as Uploaded with size 48234496 but should be 52330637, please verify the sha256 hash “iC5Pw8ClIpPuQI4T7WmInS2MfcGGmoje0BznevXiq3g=””,
“2021-09-01 13:00:01 +01 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingRemoteHash]: remote file duplicati-bbae79e428e794dc1a942e0f7954676b7.dblock.zip.aes is listed as Uploaded with size 50331648 but should be 52340797, please verify the sha256 hash “wlVt2hBtKj+6r0rZL1USZlffnwWwuxnLiYwZoeIZfxQ=””,
“2021-09-01 13:00:01 +01 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingRemoteHash]: remote file duplicati-b11911d1be9b842aa8b130cac3e029f67.dblock.zip.aes is listed as Uploaded with size 49283072 but should be 52346253, please verify the sha256 hash “smjXQq71QkBnwOSNNRwSEec4AU8UtjtTyBXVoSEf+cM=””,
“2021-09-01 13:00:01 +01 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingRemoteHash]: remote file duplicati-b8145aafea2724e20a9779231faae00b3.dblock.zip.aes is listed as Uploaded with size 48234496 but should be 52376701, please verify the sha256 hash “GB8D4x0XFAK3v7bjgfpbve4KsU/s7mttxUgkm+nogAg=””,
“2021-09-01 13:00:01 +01 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingRemoteHash]: remote file duplicati-bd5d1d13f8aae42d888284850b0a221dc.dblock.zip.aes is listed as Uploaded with size 41156608 but should be 52415709, please verify the sha256 hash “jlv98wEPoGmZyGaXxpslbQa3kIQo25wXwqBTh3n8meg=””,
“2021-09-01 13:00:01 +01 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingRemoteHash]: remote file duplicati-b63e4b178fe80410c97e4c71c61fe45ab.dblock.zip.aes is listed as Uploaded with size 29360128 but should be 52423133, please verify the sha256 hash “6AYvIzlXO90IJB+fhu01POyUuK9CKfDQ5csxY0LgyUA=””,
“2021-09-01 13:00:18 +01 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingRemoteHash]: remote file duplicati-b93a91a657a2047208b25716133b79538.dblock.zip.aes is listed as Uploaded with size 48234496 but should be 52330637, please verify the sha256 hash “iC5Pw8ClIpPuQI4T7WmInS2MfcGGmoje0BznevXiq3g=””,
“2021-09-01 13:00:18 +01 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingRemoteHash]: remote file duplicati-bbae79e428e794dc1a942e0f7954676b7.dblock.zip.aes is listed as Uploaded with size 50331648 but should be 52340797, please verify the sha256 hash “wlVt2hBtKj+6r0rZL1USZlffnwWwuxnLiYwZoeIZfxQ=””,
“2021-09-01 13:00:18 +01 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingRemoteHash]: remote file duplicati-b11911d1be9b842aa8b130cac3e029f67.dblock.zip.aes is listed as Uploaded with size 49283072 but should be 52346253, please verify the sha256 hash “smjXQq71QkBnwOSNNRwSEec4AU8UtjtTyBXVoSEf+cM=””,
“2021-09-01 13:00:18 +01 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingRemoteHash]: remote file duplicati-b8145aafea2724e20a9779231faae00b3.dblock.zip.aes is listed as Uploaded with size 48234496 but should be 52376701, please verify the sha256 hash “GB8D4x0XFAK3v7bjgfpbve4KsU/s7mttxUgkm+nogAg=””,
“2021-09-01 13:00:18 +01 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingRemoteHash]: remote file duplicati-bd5d1d13f8aae42d888284850b0a221dc.dblock.zip.aes is listed as Uploaded with size 41156608 but should be 52415709, please verify the sha256 hash “jlv98wEPoGmZyGaXxpslbQa3kIQo25wXwqBTh3n8meg=””,
“2021-09-01 13:00:18 +01 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingRemoteHash]: remote file duplicati-b63e4b178fe80410c97e4c71c61fe45ab.dblock.zip.aes is listed as Uploaded with size 29360128 but should be 52423133, please verify the sha256 hash “6AYvIzlXO90IJB+fhu01POyUuK9CKfDQ5csxY0LgyUA=””
],
“Errors”: ,
“BackendStatistics”: {
“RemoteCalls”: 9,
“BytesUploaded”: 2026,
“BytesDownloaded”: 55019271,
“FilesUploaded”: 2,
“FilesDownloaded”: 3,
“FilesDeleted”: 2,
“FoldersCreated”: 0,
“RetryAttempts”: 0,
“UnknownFileSize”: 0,
“UnknownFileCount”: 1,
“KnownFileCount”: 290,
“KnownFileSize”: 7491841132,
“LastBackupDate”: “2021-08-31T22:59:12+01:00”,
“BackupListCount”: 2,
“TotalQuotaSpace”: 3931605622784,
“FreeQuotaSpace”: 3828505280512,
“AssignedQuotaSpace”: -1,
“ReportedQuotaError”: false,
“ReportedQuotaWarning”: false,
“MainOperation”: “Backup”,
“ParsedResult”: “Success”,
“Version”: “2.0.6.3 (2.0.6.3_beta_2021-06-17)”,
“EndTime”: “0001-01-01T00:00:00”,
“BeginTime”: “2021-09-01T12:00:00.0508877Z”,
“Duration”: “00:00:00”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null
}
}

Can you provide a bit more information? It’s unclear where Duplicati is installed (on your NAS directly, or on a different machine). Are you using the NAS as a backup target, or backup source? If the NAS is the source, what is the destination and what protocol are you using?

Thanks

I have Duplicati installed on W10 Pro. I am attempting to backup from my laptop to the NAS. The connection from my laptop to my router is via WiFi and from my NAS to the router is wired CAT5 Ethernet.

I have created an account on my NAS specifically for backup and am attempting to connect to the NAS as the backup destination providing the account credentials for a NAS account within Duplicati. The source is a folder on my laptop containing many thousands of files. I am using SMB as the protocol to connect to the NAS - I believe I am using version 3. Only 11 AES files appear to be affected. The rest are OK.

Thanks - these additional details definitely give me a clearer picture of your configuration.

Did you copy the Duplicati backup files from your old NAS? Or by “new installation” do you mean this is a first run of a new backup job to a blank destination folder on the NAS?

I deleted all of the Duplicati task files from my computer. I created a brand new share on my NAS and a new backup task from scratch. It is this one that is reporting these errors but may be backing up my data. I am not making many changes at the moment.

I am not comfortable with errors being reported and will need to consider whether there are any alternative reliable backup solutions. I have used Cobian for many years and investigated Duplicati as it can backup Thunderbird mail. I have since discovered (after I had migrated all of my email from LiveMail into Thunderbird when moving into a new laptop) that Thunderbird now supports maildir format instead of mbox format giving a separate file for each email and therefore Cobian would be a viable option once I have converted my Thunderbrd mailboxes to maildir format…

My old NAS power supply failed when I disconnected it and replaced the NAS with my new one. One of the mirrored pair of drives was still working but Duplicati had failed and I was therefore unable to recover the backup. There is a previous support request on that topic but I can no longer provide further information since the NAS has died.

Ok, that’s good… I was afraid maybe you copied corrupt Duplicati data from our old NAS.

But I’m not sure why you’d be seeing this problem. Synology is pretty reliable with SMB in my experience. Maybe you can try a different protocol to see if it’s more reliable for you? SFTP and WebDAV also work really well on Synology in my experience.

If you do this you’ll want to clear out the back end files and delete the local job database to reset it to a blank state. Then you can reconfigure the Destination tab to use a different protocol and try your “first” backup again.

FYI you can look at Backup file size issue. There are plenty of tests there (and linked from it) to look at.
People only look so far (so no cause or workaround is known). It needs work by someone who sees it.

48234496 is 2E00000 in hexadecimal, so it fits the pattern of suspected filesystem or SMB issues.

I don’t know if anyone has ever tried testing with different SMB servers. It’d be interesting to know if a.Windows system as destination would have the same issue, or if Synology config is relevant to it.

It’d also be nice to see via Process Monitor (see other posts) if Duplicati wrote whole file. If so, what happened on what system to make the length wrong? Or is the length mis-reporting actual file size?
One of the linked GitHub issues found a persistent misreporting even when asking from PowerShell.

I have never succeeded in getting Duplicati to work properly. It continues to report the 12 warnings nad I have been unable to find a way round this problem.

I have now discovered that Synology has a backup solution that seems to work well enough. It may not be as economical with storage space as Duplicati but I now have plenty. I have now been using it for over a week

I am disappointed that Duplicati has failed to work as it looked like a very promising solution. I have been following Duplicati development for quite some time and I will look again from time to time.

For the record the Synology client/server uses SMB3 protocols and has been 100% reliable in backing up my data so far. Duplicati has not succeeded once without generating warnings since I moved to using the Synology NAS (my previous NAS failed suddenly) and increased the backup set to all the folders I needed. The warnings appear to be related to file size.

For now I need a solution that just works. I cannot spare any more time so I will have to uninstall Duplicati and adopt the Synology solution on both of my computers.

Yeah, I’m not sure why you are having an issue. I am unable to replicate the problem with Duplicati and my Synology NAS. I understand you not wanting to spend more time troubleshooting.

If it’s Synology “Active Backup for Business” it is a good product. I use it myself.

It is reporting errors on 11 backup files. The errors relate to a mismatch in the file size. When I first tried it on a subset of my files it worked perfectly. It was only when i asked it to backup a larger group of files that it started to show unexplained warnings. If I could sort out these reported warnings I would continue to use it.

I am however more concerned about having a reliable backup solution from my new laptop to my new NAS than diagnosing a problem in Duplicati. I have been prepared to send diagnostic information if told how to generate it and if I had the inclination I could probably dive into the code but it is over 25 years since I did any serious code debugging and I am a bit rusty! I have only dabbled with a few small C# and Python programs recently. I have used C and C++ extensively in the past (1980s)

I may look at Duplicati again in a year or two. I like the concept and have been tracking its progress now for several years.

I recently started testing Duplicati with a mult-GB backup and have several of these file size errors on my system while backing up to a NAS file share. I looked through the source code and found that the FileBackend module uses a simple stream copy with no verification at all. For such an important operation, this seems like a huge oversight by the developer.

I was able to reproduce the issue by unplugging/plugging the network cable during backup. Sometimes this does not cause an issue, sometimes the file size is different, and sometimes the files does not get created at all. There are likely many ways this issue could occur but proving that something as simple as a network interruption can corrupt a backup set is unacceptable.

I am working on adding file size validation to the FileBackend module code that would detect and retry the file creation if the file is missing or the wrong size.

1 Like

Welcome to the forum @MrWhite

With all the SMB troubles that have been reported, being able to squash problems early sounds great!
This will probably also help (probably with some performance loss) other less-than-reliable filesystems.

Were you by any chance able to use Process Monitor or strace to see if filesystem returned any error? Ignoring or mishandling those would be another way to miss a problem. Delayed errors can also occur, especially if client caching occurs. I don’t know if caching can give you a good readback then lose data.

Storage Providers shows that FTP and FTP (Alternative) do something like that. Maybe yours is similar.

Thanks!

I have not used any external tools so far. I ran the app in debug mode from VS on Windows with break points in some areas of interest.

I found that the Duplicati retry engine is quite robust but it relies on the backend module throwing an exception if a failure occurs while writing to a backup file. Sometimes the stream copy operation will throw an exception and the engine will retry and work the second time. The problem that I found is that sometimes the stream copy operation fails but does not throw an exception which the engine treats as a success.

Verifying that the backup file exists and is the expected size is quick and easy. I have already added verification code and performed several tests. My code has caught all missing and partial failures that previously went undetected. I will do some more testing to make sure it is as robust as possible then submit a PR to the main repo.

1 Like

@MrWhite thanks for your effort. It really looks that this time we will forward with this issue!