Backup showing wrong source size, huge memory use causing system crash

Not sure what changed here but have a few backups that are failing. Duplicati will fully use all 128gb of ram on server and cause server to crash.

image

Log file shows similar error, which hadn’t shown up before.

Tried restarting the app with full admin, tried rebuilding database. Not sure next step.

Hello and welcome!

Source: 563,829.21 TB

Something is definitely wrong here, unless you have an impressive NAS!

My first guess is that Duplicati got into some circular loop while walking your filesystem. Did you change any of the options regarding symlink handling? By default symlinks are stored, not followed. If you changed Duplicati to follow symlinks, and it resulted in a circular loop, I can see it causing an issue like you are seeing.

Hah, no that number is for sure wrong. I believe it said 6GB before the problem.

Not using any symlinks on this source, and advanced setting isn’t used so I assume it’s at the default (Store).

Honestly, I did not make any changes to Duplicati… but I did do an in-place upgrade recently to Windows Server (don’t judge me). I did assume the upgrade caused permission issues, but everything seems fine. I have other backups that are running fine, but it looks like the ones that are failing are all on my H:\ volume (which is a fixed disk).

I’m going to try creating a new backup for the same files and see how that goes.

Creating a new backup on the same location did not help.

image

image

image

Can you click the backup job in the main web UI and click Export. Then select “As Command-line” and uncheck “export passwords”. Click the blue Export button and paste the contents here so we can look. You can redact sensitive info of course.

I got ahead of myself.

Creating a new identical job (but pointing to a different backup destination) didn’t work.
I ended up deleting every job completely along with all backups and then restarting the service.
I created the job again, same way as the original, and now it’s working as intended.

Problem solved, but I can’t say why.

Nevermind, added second job and getting the same weird issue. It did finish properly, but it did eat memory and showed the wrong file size.

image

Export:

“C:\Program Files\Duplicati 2\Duplicati.CommandLine.exe” backup googledrive://NAS/USERS “H:\Users\” --symlink-policy=Store --snapshot-policy=Required --backup-name=NAS_USERS --dbpath=“C:\Users\Administrator\AppData\Local\Duplicati\GCSGPEVUUA.sqlite” --encryption-module=aes --compression-module=zip --dblock-size=250MB --retention-policy=“1W:1D,4W:1W,12M:1M” --disable-module=console-password-input

Log:

        {

“DeletedFiles”: 0,
“DeletedFolders”: 0,
“ModifiedFiles”: 0,
“ExaminedFiles”: 819,
“OpenedFiles”: 818,
“AddedFiles”: 818,
“SizeOfModifiedFiles”: 0,
“SizeOfAddedFiles”: 13485092821,
“SizeOfExaminedFiles”: 399415378849749,
“SizeOfOpenedFiles”: 13485092821,
“NotProcessedFiles”: 0,
“AddedFolders”: 8,
“TooLargeFiles”: 0,
“FilesWithError”: 0,
“ModifiedFolders”: 0,
“ModifiedSymlinks”: 0,
“AddedSymlinks”: 0,
“DeletedSymlinks”: 0,
“PartialBackup”: false,
“Dryrun”: false,
“MainOperation”: “Backup”,
“CompactResults”: null,
“VacuumResults”: null,
“DeleteResults”: {
“DeletedSetsActualLength”: 0,
“DeletedSets”: [],
“Dryrun”: false,
“MainOperation”: “Delete”,
“CompactResults”: null,
“ParsedResult”: “Success”,
“Version”: “2.0.6.102 (2.0.6.102_canary_2022-04-06)”,
“EndTime”: “2022-04-30T01:51:59.2591835Z”,
“BeginTime”: “2022-04-30T01:51:58.3372849Z”,
“Duration”: “00:00:00.9218986”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null,
“BackendStatistics”: {
“RemoteCalls”: 106,
“BytesUploaded”: 12960944513,
“BytesDownloaded”: 262326343,
“FilesUploaded”: 101,
“FilesDownloaded”: 3,
“FilesDeleted”: 0,
“FoldersCreated”: 0,
“RetryAttempts”: 0,
“UnknownFileSize”: 0,
“UnknownFileCount”: 0,
“KnownFileCount”: 101,
“KnownFileSize”: 12960944513,
“LastBackupDate”: “2022-04-29T20:41:11-05:00”,
“BackupListCount”: 1,
“TotalQuotaSpace”: 2199023255552,
“FreeQuotaSpace”: 2158555427209,
“AssignedQuotaSpace”: -1,
“ReportedQuotaError”: false,
“ReportedQuotaWarning”: false,
“MainOperation”: “Backup”,
“ParsedResult”: “Success”,
“Version”: “2.0.6.102 (2.0.6.102_canary_2022-04-06)”,
“EndTime”: “0001-01-01T00:00:00”,
“BeginTime”: “2022-04-30T01:41:11.1343638Z”,
“Duration”: “00:00:00”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null
}
},
“RepairResults”: null,
“TestResults”: {
“MainOperation”: “Test”,
“VerificationsActualLength”: 3,
“Verifications”: [
{
“Key”: “duplicati-20220430T014111Z.dlist.zip.aes”,
“Value”: []
},
{
“Key”: “duplicati-i61ddc6ab47bd4df1b07488701fe6cfcd.dindex.zip.aes”,
“Value”: []
},
{
“Key”: “duplicati-b1ae01f361d084321bcc93a6ad5b00233.dblock.zip.aes”,
“Value”: []
}
],
“ParsedResult”: “Success”,
“Version”: “2.0.6.102 (2.0.6.102_canary_2022-04-06)”,
“EndTime”: “2022-04-30T01:52:07.79071Z”,
“BeginTime”: “2022-04-30T01:52:01.6655169Z”,
“Duration”: “00:00:06.1251931”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null,
“BackendStatistics”: {
“RemoteCalls”: 106,
“BytesUploaded”: 12960944513,
“BytesDownloaded”: 262326343,
“FilesUploaded”: 101,
“FilesDownloaded”: 3,
“FilesDeleted”: 0,
“FoldersCreated”: 0,
“RetryAttempts”: 0,
“UnknownFileSize”: 0,
“UnknownFileCount”: 0,
“KnownFileCount”: 101,
“KnownFileSize”: 12960944513,
“LastBackupDate”: “2022-04-29T20:41:11-05:00”,
“BackupListCount”: 1,
“TotalQuotaSpace”: 2199023255552,
“FreeQuotaSpace”: 2158555427209,
“AssignedQuotaSpace”: -1,
“ReportedQuotaError”: false,
“ReportedQuotaWarning”: false,
“MainOperation”: “Backup”,
“ParsedResult”: “Success”,
“Version”: “2.0.6.102 (2.0.6.102_canary_2022-04-06)”,
“EndTime”: “0001-01-01T00:00:00”,
“BeginTime”: “2022-04-30T01:41:11.1343638Z”,
“Duration”: “00:00:00”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null
}
},
“ParsedResult”: “Warning”,
“Version”: “2.0.6.102 (2.0.6.102_canary_2022-04-06)”,
“EndTime”: “2022-04-30T01:52:07.8063022Z”,
“BeginTime”: “2022-04-30T01:41:11.1343638Z”,
“Duration”: “00:10:56.6719384”,
“MessagesActualLength”: 219,
“WarningsActualLength”: 1,
“ErrorsActualLength”: 0,
“Messages”: [
“2022-04-29 20:41:11 -05 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started”,
“2022-04-29 20:41:18 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: ()”,
“2022-04-29 20:41:19 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: ()”,
“2022-04-29 20:44:12 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-be7deb145f9e84f5c942b750e918f55f5.dblock.zip.aes (249.92 MB)”,
“2022-04-29 20:44:28 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-be7deb145f9e84f5c942b750e918f55f5.dblock.zip.aes (249.92 MB)”,
“2022-04-29 20:44:28 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b8cc779187a334d398684604927f75f43.dblock.zip.aes (249.93 MB)”,
“2022-04-29 20:44:28 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b056245cd136146d88d80c3c474a22ef6.dblock.zip.aes (249.94 MB)”,
“2022-04-29 20:44:28 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b0f98d1078768438e9120c3015735eefb.dblock.zip.aes (249.93 MB)”,
“2022-04-29 20:44:32 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-ib35d63bd20fe40f7bcf0e5b1032e62ba.dindex.zip.aes (215.03 KB)”,
“2022-04-29 20:44:33 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-ib35d63bd20fe40f7bcf0e5b1032e62ba.dindex.zip.aes (215.03 KB)”,
“2022-04-29 20:44:33 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b09b78a07b52f40e2a87fc42ba9090fd8.dblock.zip.aes (249.93 MB)”,
“2022-04-29 20:44:48 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b09b78a07b52f40e2a87fc42ba9090fd8.dblock.zip.aes (249.93 MB)”,
“2022-04-29 20:44:59 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b0f98d1078768438e9120c3015735eefb.dblock.zip.aes (249.93 MB)”,
“2022-04-29 20:44:59 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b056245cd136146d88d80c3c474a22ef6.dblock.zip.aes (249.94 MB)”,
“2022-04-29 20:45:04 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b8cc779187a334d398684604927f75f43.dblock.zip.aes (249.93 MB)”,
“2022-04-29 20:45:04 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-i8d096a48748e423f80a118fada2f057b.dindex.zip.aes (122.23 KB)”,
“2022-04-29 20:45:06 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-i8d096a48748e423f80a118fada2f057b.dindex.zip.aes (122.23 KB)”,
“2022-04-29 20:45:06 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-i61ddc6ab47bd4df1b07488701fe6cfcd.dindex.zip.aes (191.92 KB)”,
“2022-04-29 20:45:06 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-i5c8ef2a4cf8f418dae329ff6e88e701f.dindex.zip.aes (214.53 KB)”,
“2022-04-29 20:45:08 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-i5c8ef2a4cf8f418dae329ff6e88e701f.dindex.zip.aes (214.53 KB)”
],
“Warnings”: [
“2022-04-29 20:41:20 -05 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: H:\Users\Tarv\OfficeUninstall.diagcab”
],
“Errors”: [],
“BackendStatistics”: {
“RemoteCalls”: 106,
“BytesUploaded”: 12960944513,
“BytesDownloaded”: 262326343,
“FilesUploaded”: 101,
“FilesDownloaded”: 3,
“FilesDeleted”: 0,
“FoldersCreated”: 0,
“RetryAttempts”: 0,
“UnknownFileSize”: 0,
“UnknownFileCount”: 0,
“KnownFileCount”: 101,
“KnownFileSize”: 12960944513,
“LastBackupDate”: “2022-04-29T20:41:11-05:00”,
“BackupListCount”: 1,
“TotalQuotaSpace”: 2199023255552,
“FreeQuotaSpace”: 2158555427209,
“AssignedQuotaSpace”: -1,
“ReportedQuotaError”: false,
“ReportedQuotaWarning”: false,
“MainOperation”: “Backup”,
“ParsedResult”: “Success”,
“Version”: “2.0.6.102 (2.0.6.102_canary_2022-04-06)”,
“EndTime”: “0001-01-01T00:00:00”,
“BeginTime”: “2022-04-30T01:41:11.1343638Z”,
“Duration”: “00:00:00”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null
}
}