Purging backup of warning generating files?

Pichu0102 · July 1, 2019, 10:46am

I’m having an issue where I get daily warnings on my backup. I’ve started to get lazy and not check daily from all the false alarms (I assume) when it pops up the warning message. That’s not good.
I’ve been trying to purge the offending filenames from the backups but all the purge dry run attempts end up purging all or no files from the backup.

Example error from the backup task:

[details=“Summary”]
DeletedFiles: 5938
DeletedFolders: 421
ModifiedFiles: 4287
ExaminedFiles: 533596
OpenedFiles: 8222
AddedFiles: 3935
SizeOfModifiedFiles: 41796891334
SizeOfAddedFiles: 835682566
SizeOfExaminedFiles: 1535936398773
SizeOfOpenedFiles: 42791837266
NotProcessedFiles: 0
AddedFolders: 30
TooLargeFiles: 0
FilesWithError: 0
ModifiedFolders: 0
ModifiedSymlinks: 0
AddedSymlinks: 0
DeletedSymlinks: 0
PartialBackup: False
Dryrun: False
MainOperation: Backup
CompactResults:
DeletedFileCount: 0
DownloadedFileCount: 0
UploadedFileCount: 0
DeletedFileSize: 0
DownloadedFileSize: 0
UploadedFileSize: 0
Dryrun: False
MainOperation: Compact
ParsedResult: Warning
Version: 2.0.4.5 (2.0.4.5_beta_2018-11-28)
EndTime: 7/1/2019 3:43:38 AM (1561967018)
BeginTime: 7/1/2019 3:37:48 AM (1561966668)
Duration: 00:05:50.4142798
Messages: [
2019-07-01 01:00:01 -04 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started,
2019-07-01 01:18:02 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: (),
2019-07-01 01:19:32 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: (5.02 KB),
2019-07-01 02:56:19 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-ba0150b55b8694953b5bb86a44e6352e9.dblock.zip (499.97 MB),
2019-07-01 02:58:35 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-ba0150b55b8694953b5bb86a44e6352e9.dblock.zip (499.97 MB),
…
]
Warnings: [
2019-07-01 01:20:42 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\Users\Pichu0102\AppData\Local\NVIDIA Corporation\NVIDIA Share\CefCache\LOCK,
2019-07-01 01:20:42 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\Users\Pichu0102\AppData\Local\NVIDIA Corporation\NVIDIA Share\CefCache\Local Storage\leveldb\LOCK,
2019-07-01 01:20:42 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\Users\Pichu0102\AppData\Local\NVIDIA Corporation\NVIDIA Share\CefCache\IndexedDB\file__0.indexeddb.leveldb\LOCK,
2019-07-01 01:20:42 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\Users\Pichu0102\AppData\Local\NVIDIA Corporation\NVIDIA Share\CefCache\GPUCache\data_0,
2019-07-01 01:20:42 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\Users\Pichu0102\AppData\Local\NVIDIA Corporation\NVIDIA Share\CefCache\GPUCache\data_1,
…
]
Errors:
BackendStatistics:
RemoteCalls: 17
BytesUploaded: 2365994101
BytesDownloaded: 581226909
FilesUploaded: 11
FilesDownloaded: 3
FilesDeleted: 1
FoldersCreated: 0
RetryAttempts: 0
UnknownFileSize: 0
UnknownFileCount: 0
KnownFileCount: 5155
KnownFileSize: 1339875664260
LastBackupDate: 7/1/2019 1:00:01 AM (1561957201)
BackupListCount: 9
TotalQuotaSpace: 20253444628480
FreeQuotaSpace: 6718944612352
AssignedQuotaSpace: -1
ReportedQuotaError: False
ReportedQuotaWarning: False
ParsedResult: Warning
Version: 2.0.4.5 (2.0.4.5_beta_2018-11-28)
Messages: [
2019-07-01 01:00:01 -04 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started,
2019-07-01 01:18:02 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: (),
2019-07-01 01:19:32 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: (5.02 KB),
2019-07-01 02:56:19 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-ba0150b55b8694953b5bb86a44e6352e9.dblock.zip (499.97 MB),
2019-07-01 02:58:35 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-ba0150b55b8694953b5bb86a44e6352e9.dblock.zip (499.97 MB),
…
]
Warnings: [
2019-07-01 01:20:42 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\Users\Pichu0102\AppData\Local\NVIDIA Corporation\NVIDIA Share\CefCache\LOCK,
2019-07-01 01:20:42 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\Users\Pichu0102\AppData\Local\NVIDIA Corporation\NVIDIA Share\CefCache\Local Storage\leveldb\LOCK,
2019-07-01 01:20:42 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\Users\Pichu0102\AppData\Local\NVIDIA Corporation\NVIDIA Share\CefCache\IndexedDB\file__0.indexeddb.leveldb\LOCK,
2019-07-01 01:20:42 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\Users\Pichu0102\AppData\Local\NVIDIA Corporation\NVIDIA Share\CefCache\GPUCache\data_0,
2019-07-01 01:20:42 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\Users\Pichu0102\AppData\Local\NVIDIA Corporation\NVIDIA Share\CefCache\GPUCache\data_1,
…
]
Errors:
DeleteResults:
DeletedSets: [
Item1: 5
Item2: 6/24/2019 2:44:55 AM (1561358695)
]
Dryrun: False
MainOperation: Delete
ParsedResult: Warning
Version: 2.0.4.5 (2.0.4.5_beta_2018-11-28)
EndTime: 7/1/2019 3:43:38 AM (1561967018)
BeginTime: 7/1/2019 3:28:14 AM (1561966094)
Duration: 00:15:24.5073633
Messages: [
2019-07-01 01:00:01 -04 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started,
2019-07-01 01:18:02 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: (),
2019-07-01 01:19:32 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: (5.02 KB),
2019-07-01 02:56:19 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-ba0150b55b8694953b5bb86a44e6352e9.dblock.zip (499.97 MB),
2019-07-01 02:58:35 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-ba0150b55b8694953b5bb86a44e6352e9.dblock.zip (499.97 MB),
…
]
Warnings: [
2019-07-01 01:20:42 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\Users\Pichu0102\AppData\Local\NVIDIA Corporation\NVIDIA Share\CefCache\LOCK,
2019-07-01 01:20:42 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\Users\Pichu0102\AppData\Local\NVIDIA Corporation\NVIDIA Share\CefCache\Local Storage\leveldb\LOCK,
2019-07-01 01:20:42 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\Users\Pichu0102\AppData\Local\NVIDIA Corporation\NVIDIA Share\CefCache\IndexedDB\file__0.indexeddb.leveldb\LOCK,
2019-07-01 01:20:42 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\Users\Pichu0102\AppData\Local\NVIDIA Corporation\NVIDIA Share\CefCache\GPUCache\data_0,
2019-07-01 01:20:42 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\Users\Pichu0102\AppData\Local\NVIDIA Corporation\NVIDIA Share\CefCache\GPUCache\data_1,
…
]
Errors:
RepairResults: null
TestResults:
MainOperation: Test
Verifications: [
Key: duplicati-20190701T050001Z.dlist.zip
Value: ,
Key: duplicati-iea308fc1ee9b4e6c8b56df07fdf07921.dindex.zip
Value: ,
Key: duplicati-b0b6baebb553a447791848ff912be8053.dblock.zip
Value:
]
ParsedResult: Warning
Version: 2.0.4.5 (2.0.4.5_beta_2018-11-28)
EndTime: 7/1/2019 3:44:23 AM (1561967063)
BeginTime: 7/1/2019 3:44:18 AM (1561967058)
Duration: 00:00:05.1251279
Messages: [
2019-07-01 01:00:01 -04 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started,
2019-07-01 01:18:02 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: (),
2019-07-01 01:19:32 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: (5.02 KB),
2019-07-01 02:56:19 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-ba0150b55b8694953b5bb86a44e6352e9.dblock.zip (499.97 MB),
2019-07-01 02:58:35 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-ba0150b55b8694953b5bb86a44e6352e9.dblock.zip (499.97 MB),
…
]
Warnings: [
2019-07-01 01:20:42 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\Users\Pichu0102\AppData\Local\NVIDIA Corporation\NVIDIA Share\CefCache\LOCK,
2019-07-01 01:20:42 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\Users\Pichu0102\AppData\Local\NVIDIA Corporation\NVIDIA Share\CefCache\Local Storage\leveldb\LOCK,
2019-07-01 01:20:42 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\Users\Pichu0102\AppData\Local\NVIDIA Corporation\NVIDIA Share\CefCache\IndexedDB\file__0.indexeddb.leveldb\LOCK,
2019-07-01 01:20:42 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\Users\Pichu0102\AppData\Local\NVIDIA Corporation\NVIDIA Share\CefCache\GPUCache\data_0,
2019-07-01 01:20:42 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\Users\Pichu0102\AppData\Local\NVIDIA Corporation\NVIDIA Share\CefCache\GPUCache\data_1,
…
]
Errors:
ParsedResult: Warning
Version: 2.0.4.5 (2.0.4.5_beta_2018-11-28)
EndTime: 7/1/2019 3:44:23 AM (1561967063)
BeginTime: 7/1/2019 1:00:00 AM (1561957200)
Duration: 02:44:23.2846262
Messages: [
2019-07-01 01:00:01 -04 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started,
2019-07-01 01:18:02 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: (),
2019-07-01 01:19:32 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: (5.02 KB),
2019-07-01 02:56:19 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-ba0150b55b8694953b5bb86a44e6352e9.dblock.zip (499.97 MB),
2019-07-01 02:58:35 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-ba0150b55b8694953b5bb86a44e6352e9.dblock.zip (499.97 MB),
…
]
Warnings: [
2019-07-01 01:20:42 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\Users\Pichu0102\AppData\Local\NVIDIA Corporation\NVIDIA Share\CefCache\LOCK,
2019-07-01 01:20:42 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\Users\Pichu0102\AppData\Local\NVIDIA Corporation\NVIDIA Share\CefCache\Local Storage\leveldb\LOCK,
2019-07-01 01:20:42 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\Users\Pichu0102\AppData\Local\NVIDIA Corporation\NVIDIA Share\CefCache\IndexedDB\file__0.indexeddb.leveldb\LOCK,
2019-07-01 01:20:42 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\Users\Pichu0102\AppData\Local\NVIDIA Corporation\NVIDIA Share\CefCache\GPUCache\data_0,
2019-07-01 01:20:42 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\Users\Pichu0102\AppData\Local\NVIDIA Corporation\NVIDIA Share\CefCache\GPUCache\data_1,
…
]
Errors:
[/details]

drwtsn32 · July 1, 2019, 3:25pm

The problem is you are trying to back up locked files.

You can either deselect those folders from the backup selection list (no need to try and purge them out of past backups), or you could enable the --snapshot-policy option.

Regarding option 1 - in most cases programs do not store useful user data in AppData\Local, so it’s not worth backing up. (AppData\Roaming on the other hand should be backed up.)

Regarding option 2 - To enable the snapshot option you’ll need to make sure the Duplicati process runs with elevated Administrative rights. There are a couple ways to accomplish that. Running it as a service is one option, but it’s not completely straight forward - you may need to relocate Duplicati data files. I believe there is a how-to posted in the forum.

Pichu0102 · July 12, 2019, 4:46am

I have removed the files from the backup selection list, but the errors still occur. I have Duplicati’s tray icon start from a Windows scheduled task to run with highest privileges. I have snapshot policy set to Auto, since I also back up a pooled drive that doesn’t support VSS, but that isn’t the drive the errored files reside on. And unfortunately, one too many games do like to save to Local and LocalLow to filter them out entirely.

ts678 · July 12, 2019, 7:27pm

You can look into what’s happening (or not happening) with –log-file and –log-file-log-level=verbose, to see how it’s handling your files. Here’s what 2.0.4.22 canary said about Duplicati’s own lock file when I backed up its folder, first with an un-check on the lock_v2 file, then without (which, as expected, raised a warning):

2019-07-12 15:04:08 -04 - [Verbose-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-ExcludingPathFromFilter]: Excluding path due to filter: C:\ProgramData\Duplicati\duplicati-2.0.4.22_canary_2019-06-30\RUN\control_dir_v2\lock_v2 => (C:\ProgramData\Duplicati\duplicati-2.0.4.22_canary_2019-06-30\RUN\control_dir_v2\lock_v2)
2019-07-12 15:04:08 -04 - [Verbose-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-ExcludingPathFromFilter]: Excluding path due to filter: C:\ProgramData\Duplicati\duplicati-2.0.4.22_canary_2019-06-30\RUN\control_dir_v2\lock_v2 => (C:\ProgramData\Duplicati\duplicati-2.0.4.22_canary_2019-06-30\RUN\control_dir_v2\lock_v2)
2019-07-12 15:08:03 -04 - [Verbose-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-IncludingPath]: Including path as no filters matched: C:\ProgramData\Duplicati\duplicati-2.0.4.22_canary_2019-06-30\RUN\control_dir_v2\lock_v2
2019-07-12 15:08:03 -04 - [Verbose-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-IncludingPath]: Including path as no filters matched: C:\ProgramData\Duplicati\duplicati-2.0.4.22_canary_2019-06-30\RUN\control_dir_v2\lock_v2
2019-07-12 15:08:03 -04 - [Verbose-Duplicati.Library.Main.Operation.Backup.FilePreFilterProcess.FileEntry-CheckFileForChanges]: Checking file for changes C:\ProgramData\Duplicati\duplicati-2.0.4.22_canary_2019-06-30\RUN\control_dir_v2\lock_v2, new: True, timestamp changed: True, size changed: True, metadatachanged: True, 7/12/2019 12:41:22 AM vs 1/1/0001 12:00:00 AM
2019-07-12 15:08:03 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\ProgramData\Duplicati\duplicati-2.0.4.22_canary_2019-06-30\RUN\control_dir_v2\lock_v2
System.IO.IOException: The process cannot access the file 'C:\ProgramData\Duplicati\duplicati-2.0.4.22_canary_2019-06-30\RUN\control_dir_v2\lock_v2' because it is being used by another process.

If you want to get extra tricky (and get a smaller log), try –log-file-log-filter, otherwise just search the log file.

EDIT:

I wasn’t sure how an un-check was represented in the options syntax (which is probably also how the GUI remembers it) but running Export as Command-line found it generates an –exclude for the un-checked file, and this fits with the language in the log. One can also see the Filter in the Filters area of Source data step.