Not sure what changed here but have a few backups that are failing. Duplicati will fully use all 128gb of ram on server and cause server to crash.
Log file shows similar error, which hadn’t shown up before.
Tried restarting the app with full admin, tried rebuilding database. Not sure next step.
Hello and welcome!
Source: 563,829.21 TB
Something is definitely wrong here, unless you have an impressive NAS!
My first guess is that Duplicati got into some circular loop while walking your filesystem. Did you change any of the options regarding symlink handling? By default symlinks are stored, not followed. If you changed Duplicati to follow symlinks, and it resulted in a circular loop, I can see it causing an issue like you are seeing.
Hah, no that number is for sure wrong. I believe it said 6GB before the problem.
Not using any symlinks on this source, and advanced setting isn’t used so I assume it’s at the default (Store).
Honestly, I did not make any changes to Duplicati… but I did do an in-place upgrade recently to Windows Server (don’t judge me). I did assume the upgrade caused permission issues, but everything seems fine. I have other backups that are running fine, but it looks like the ones that are failing are all on my H:\ volume (which is a fixed disk).
I’m going to try creating a new backup for the same files and see how that goes.
Creating a new backup on the same location did not help.
Can you click the backup job in the main web UI and click Export. Then select “As Command-line” and uncheck “export passwords”. Click the blue Export button and paste the contents here so we can look. You can redact sensitive info of course.
I got ahead of myself.
Creating a new identical job (but pointing to a different backup destination) didn’t work.
I ended up deleting every job completely along with all backups and then restarting the service.
I created the job again, same way as the original, and now it’s working as intended.
Problem solved, but I can’t say why.
Nevermind, added second job and getting the same weird issue. It did finish properly, but it did eat memory and showed the wrong file size.
Export:
“C:\Program Files\Duplicati 2\Duplicati.CommandLine.exe” backup googledrive://NAS/USERS “H:\Users\” --symlink-policy=Store --snapshot-policy=Required --backup-name=NAS_USERS --dbpath=“C:\Users\Administrator\AppData\Local\Duplicati\GCSGPEVUUA.sqlite” --encryption-module=aes --compression-module=zip --dblock-size=250MB --retention-policy=“1W:1D,4W:1W,12M:1M” --disable-module=console-password-input
Log:
{“DeletedFiles”: 0,
“DeletedFolders”: 0,
“ModifiedFiles”: 0,
“ExaminedFiles”: 819,
“OpenedFiles”: 818,
“AddedFiles”: 818,
“SizeOfModifiedFiles”: 0,
“SizeOfAddedFiles”: 13485092821,
“SizeOfExaminedFiles”: 399415378849749,
“SizeOfOpenedFiles”: 13485092821,
“NotProcessedFiles”: 0,
“AddedFolders”: 8,
“TooLargeFiles”: 0,
“FilesWithError”: 0,
“ModifiedFolders”: 0,
“ModifiedSymlinks”: 0,
“AddedSymlinks”: 0,
“DeletedSymlinks”: 0,
“PartialBackup”: false,
“Dryrun”: false,
“MainOperation”: “Backup”,
“CompactResults”: null,
“VacuumResults”: null,
“DeleteResults”: {
“DeletedSetsActualLength”: 0,
“DeletedSets”: ,
“Dryrun”: false,
“MainOperation”: “Delete”,
“CompactResults”: null,
“ParsedResult”: “Success”,
“Version”: “2.0.6.102 (2.0.6.102_canary_2022-04-06)”,
“EndTime”: “2022-04-30T01:51:59.2591835Z”,
“BeginTime”: “2022-04-30T01:51:58.3372849Z”,
“Duration”: “00:00:00.9218986”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null,
“BackendStatistics”: {
“RemoteCalls”: 106,
“BytesUploaded”: 12960944513,
“BytesDownloaded”: 262326343,
“FilesUploaded”: 101,
“FilesDownloaded”: 3,
“FilesDeleted”: 0,
“FoldersCreated”: 0,
“RetryAttempts”: 0,
“UnknownFileSize”: 0,
“UnknownFileCount”: 0,
“KnownFileCount”: 101,
“KnownFileSize”: 12960944513,
“LastBackupDate”: “2022-04-29T20:41:11-05:00”,
“BackupListCount”: 1,
“TotalQuotaSpace”: 2199023255552,
“FreeQuotaSpace”: 2158555427209,
“AssignedQuotaSpace”: -1,
“ReportedQuotaError”: false,
“ReportedQuotaWarning”: false,
“MainOperation”: “Backup”,
“ParsedResult”: “Success”,
“Version”: “2.0.6.102 (2.0.6.102_canary_2022-04-06)”,
“EndTime”: “0001-01-01T00:00:00”,
“BeginTime”: “2022-04-30T01:41:11.1343638Z”,
“Duration”: “00:00:00”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null
}
},
“RepairResults”: null,
“TestResults”: {
“MainOperation”: “Test”,
“VerificationsActualLength”: 3,
“Verifications”: [
{
“Key”: “duplicati-20220430T014111Z.dlist.zip.aes”,
“Value”:
},
{
“Key”: “duplicati-i61ddc6ab47bd4df1b07488701fe6cfcd.dindex.zip.aes”,
“Value”:
},
{
“Key”: “duplicati-b1ae01f361d084321bcc93a6ad5b00233.dblock.zip.aes”,
“Value”:
}
],
“ParsedResult”: “Success”,
“Version”: “2.0.6.102 (2.0.6.102_canary_2022-04-06)”,
“EndTime”: “2022-04-30T01:52:07.79071Z”,
“BeginTime”: “2022-04-30T01:52:01.6655169Z”,
“Duration”: “00:00:06.1251931”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null,
“BackendStatistics”: {
“RemoteCalls”: 106,
“BytesUploaded”: 12960944513,
“BytesDownloaded”: 262326343,
“FilesUploaded”: 101,
“FilesDownloaded”: 3,
“FilesDeleted”: 0,
“FoldersCreated”: 0,
“RetryAttempts”: 0,
“UnknownFileSize”: 0,
“UnknownFileCount”: 0,
“KnownFileCount”: 101,
“KnownFileSize”: 12960944513,
“LastBackupDate”: “2022-04-29T20:41:11-05:00”,
“BackupListCount”: 1,
“TotalQuotaSpace”: 2199023255552,
“FreeQuotaSpace”: 2158555427209,
“AssignedQuotaSpace”: -1,
“ReportedQuotaError”: false,
“ReportedQuotaWarning”: false,
“MainOperation”: “Backup”,
“ParsedResult”: “Success”,
“Version”: “2.0.6.102 (2.0.6.102_canary_2022-04-06)”,
“EndTime”: “0001-01-01T00:00:00”,
“BeginTime”: “2022-04-30T01:41:11.1343638Z”,
“Duration”: “00:00:00”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null
}
},
“ParsedResult”: “Warning”,
“Version”: “2.0.6.102 (2.0.6.102_canary_2022-04-06)”,
“EndTime”: “2022-04-30T01:52:07.8063022Z”,
“BeginTime”: “2022-04-30T01:41:11.1343638Z”,
“Duration”: “00:10:56.6719384”,
“MessagesActualLength”: 219,
“WarningsActualLength”: 1,
“ErrorsActualLength”: 0,
“Messages”: [
“2022-04-29 20:41:11 -05 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started”,
“2022-04-29 20:41:18 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: ()”,
“2022-04-29 20:41:19 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: ()”,
“2022-04-29 20:44:12 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-be7deb145f9e84f5c942b750e918f55f5.dblock.zip.aes (249.92 MB)”,
“2022-04-29 20:44:28 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-be7deb145f9e84f5c942b750e918f55f5.dblock.zip.aes (249.92 MB)”,
“2022-04-29 20:44:28 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b8cc779187a334d398684604927f75f43.dblock.zip.aes (249.93 MB)”,
“2022-04-29 20:44:28 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b056245cd136146d88d80c3c474a22ef6.dblock.zip.aes (249.94 MB)”,
“2022-04-29 20:44:28 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b0f98d1078768438e9120c3015735eefb.dblock.zip.aes (249.93 MB)”,
“2022-04-29 20:44:32 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-ib35d63bd20fe40f7bcf0e5b1032e62ba.dindex.zip.aes (215.03 KB)”,
“2022-04-29 20:44:33 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-ib35d63bd20fe40f7bcf0e5b1032e62ba.dindex.zip.aes (215.03 KB)”,
“2022-04-29 20:44:33 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b09b78a07b52f40e2a87fc42ba9090fd8.dblock.zip.aes (249.93 MB)”,
“2022-04-29 20:44:48 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b09b78a07b52f40e2a87fc42ba9090fd8.dblock.zip.aes (249.93 MB)”,
“2022-04-29 20:44:59 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b0f98d1078768438e9120c3015735eefb.dblock.zip.aes (249.93 MB)”,
“2022-04-29 20:44:59 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b056245cd136146d88d80c3c474a22ef6.dblock.zip.aes (249.94 MB)”,
“2022-04-29 20:45:04 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b8cc779187a334d398684604927f75f43.dblock.zip.aes (249.93 MB)”,
“2022-04-29 20:45:04 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-i8d096a48748e423f80a118fada2f057b.dindex.zip.aes (122.23 KB)”,
“2022-04-29 20:45:06 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-i8d096a48748e423f80a118fada2f057b.dindex.zip.aes (122.23 KB)”,
“2022-04-29 20:45:06 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-i61ddc6ab47bd4df1b07488701fe6cfcd.dindex.zip.aes (191.92 KB)”,
“2022-04-29 20:45:06 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-i5c8ef2a4cf8f418dae329ff6e88e701f.dindex.zip.aes (214.53 KB)”,
“2022-04-29 20:45:08 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-i5c8ef2a4cf8f418dae329ff6e88e701f.dindex.zip.aes (214.53 KB)”
],
“Warnings”: [
“2022-04-29 20:41:20 -05 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: H:\Users\Tarv\OfficeUninstall.diagcab”
],
“Errors”: ,
“BackendStatistics”: {
“RemoteCalls”: 106,
“BytesUploaded”: 12960944513,
“BytesDownloaded”: 262326343,
“FilesUploaded”: 101,
“FilesDownloaded”: 3,
“FilesDeleted”: 0,
“FoldersCreated”: 0,
“RetryAttempts”: 0,
“UnknownFileSize”: 0,
“UnknownFileCount”: 0,
“KnownFileCount”: 101,
“KnownFileSize”: 12960944513,
“LastBackupDate”: “2022-04-29T20:41:11-05:00”,
“BackupListCount”: 1,
“TotalQuotaSpace”: 2199023255552,
“FreeQuotaSpace”: 2158555427209,
“AssignedQuotaSpace”: -1,
“ReportedQuotaError”: false,
“ReportedQuotaWarning”: false,
“MainOperation”: “Backup”,
“ParsedResult”: “Success”,
“Version”: “2.0.6.102 (2.0.6.102_canary_2022-04-06)”,
“EndTime”: “0001-01-01T00:00:00”,
“BeginTime”: “2022-04-30T01:41:11.1343638Z”,
“Duration”: “00:00:00”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null
}
}
Hey, I’m getting the same thing. Its ~10GB but showing 128TB. Did anyone get back to you about this?
Sorry, don’t recall what the resolution was. I seem to recall just recreating all my jobs and starting from scratch.
and interestingly you’re on a different OS, however both have Sparse file support which can make files larger than the space they actually consume. Were you watching the status bar for a sudden big jump? Does the file in the screenshot that’s 0.00% processed have any significance, or is it just random shot?
There are a lot of methods to pursue this. Easiest is to open database copy with sqlitebrowser, but not everyone’s willing and able to do that. Setting up log files may be another option to look for a huge file.
Linux tools such as du ls and find (or whatever your file manager has) can also check out the file sizes.
and something similar to narrow down this issue would be to run different subset jobs to find the culprit.
EDIT:
Of course some of these methods work best if it finally completes, but if it doesn’t there are other ways depending on which tools you prefer. If the GUI shows it on the same file, look over that file using tools. Linux tools are good. Duplicati’s advanced live tools include About → Show log → Live to watch what’s going on, and About → System info → Server state properties → lastPgEvent for detailed inside clues.
