Since a few weeks I am getting these warnings on my daily backup on some days (not every day). The warnings are always related to Prometheus, which volume I am backupping on a daily base.
here is the warning:
LimitedWarnings: [
2023-09-24 03:00:06 +02 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: /source/var/lib/docker/volumes/prometheus-data/_data/data/wal/checkpoint.00000182/,
2023-09-24 03:00:06 +02 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: /source/var/lib/docker/volumes/prometheus-data/_data/data/wal/00000184,
2023-09-24 03:00:06 +02 - [Warning-Duplicati.Library.Main.Operation.Backup.MetadataGenerator.Metadata-MetadataProcessFailed]: Failed to process metadata for "/source/var/lib/docker/volumes/prometheus-data/_data/data/wal/00000183", storing empty metadata,
2023-09-24 03:00:06 +02 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-PathProcessingError]: Failed to process path: /source/var/lib/docker/volumes/prometheus-data/_data/data/wal/checkpoint.00000182/,
2023-09-24 03:00:06 +02 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: /source/var/lib/docker/volumes/prometheus-data/_data/data/wal/checkpoint.00000182/,
2023-09-24 03:00:06 +02 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: /source/var/lib/docker/volumes/prometheus-data/_data/data/wal/00000183,
2023-09-24 03:00:06 +02 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-PathProcessingError]: Failed to process path: /source/var/lib/docker/volumes/prometheus-data/_data/data/wal/checkpoint.00000182/,
2023-09-24 03:00:06 +02 - [Warning-Duplicati.Library.Main.Operation.Backup.MetadataGenerator.Metadata-MetadataProcessFailed]: Failed to process metadata for "/source/var/lib/docker/volumes/prometheus-data/_data/data/wal/checkpoint.00000182/", storing empty metadata,
2023-09-24 03:00:06 +02 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: /source/var/lib/docker/volumes/prometheus-data/_data/data/wal/checkpoint.00000182/
]
Anyone experiencing the same issue? How can I solve this?
is there any chance that these files could be locked by a different process / container ? Normally open files can be accessed but where container are concerned there can be weird operating system stuff going on.
OK. So what would you suggest to prevent these warnings? Is there a way to ignore these files or is it a good idea to just turn of the notifications for Warnings?
On-disk layout supports that idea, but just like SQLite, Deleting a hot journal can lose information. Compaction sounds like what periodically gets the raw data out of the write-ahead log files. IDK…
Possibly it’s related to compaction or some other activity done on wal files. The checkpoint folders probably won’t suffer from locking, but I’m not sure what they are, and whether they come and go.
If live log is too slow, you can let log-file=<path> using log-file-log-filter=warning monitor for awhile.
I think there’s a pull request to put a little more information in the regular log on the specific cause.
There’s generally a deeper question for databases or any complex layout that changes under you. Making the files stand still (or give a static snapshot) is necessary to get a consistent files backup.
Generally one should look for some database-specific advice for backup and restore. For example: Snapshots, backup and restore #3838 which just pointed to their mailing list – you can check there. Feedback on time series backup strategy. How do you do it? which links to a few things people run. Prometheus export / import data for backup and similar ideas that one can find with Internet search.
This page had one rather gloomy view of the situation. You can decide how serious you want to be.
Even a semi-serious backup deserves restore tests, so there’s something else you should consider.
Reading your answer leads me to the question if making backups of Prometheus in general makes sense … But I think I will just try to ignore warnings from Prometheus volumes …
Although it was a 2017 answer, that was the gloomy view – and accepted answer at Stack Overflow.
Choosing to not back it up will certainly give you a lot of room to exclude your way past its warnings.