Warnings when making a backup of Prometheus volume

Support
1

Since a few weeks I am getting these warnings on my daily backup on some days (not every day). The warnings are always related to Prometheus, which volume I am backupping on a daily base.

here is the warning:

LimitedWarnings: [
   2023-09-24 03:00:06 +02 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: /source/var/lib/docker/volumes/prometheus-data/_data/data/wal/checkpoint.00000182/,
   2023-09-24 03:00:06 +02 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: /source/var/lib/docker/volumes/prometheus-data/_data/data/wal/00000184,
   2023-09-24 03:00:06 +02 - [Warning-Duplicati.Library.Main.Operation.Backup.MetadataGenerator.Metadata-MetadataProcessFailed]: Failed to process metadata for "/source/var/lib/docker/volumes/prometheus-data/_data/data/wal/00000183", storing empty metadata,
   2023-09-24 03:00:06 +02 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-PathProcessingError]: Failed to process path: /source/var/lib/docker/volumes/prometheus-data/_data/data/wal/checkpoint.00000182/,
   2023-09-24 03:00:06 +02 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: /source/var/lib/docker/volumes/prometheus-data/_data/data/wal/checkpoint.00000182/,
   2023-09-24 03:00:06 +02 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: /source/var/lib/docker/volumes/prometheus-data/_data/data/wal/00000183,
   2023-09-24 03:00:06 +02 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-PathProcessingError]: Failed to process path: /source/var/lib/docker/volumes/prometheus-data/_data/data/wal/checkpoint.00000182/,
   2023-09-24 03:00:06 +02 - [Warning-Duplicati.Library.Main.Operation.Backup.MetadataGenerator.Metadata-MetadataProcessFailed]: Failed to process metadata for "/source/var/lib/docker/volumes/prometheus-data/_data/data/wal/checkpoint.00000182/", storing empty metadata,
   2023-09-24 03:00:06 +02 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: /source/var/lib/docker/volumes/prometheus-data/_data/data/wal/checkpoint.00000182/
]

Anyone experiencing the same issue? How can I solve this?

2

Hello

is there any chance that these files could be locked by a different process / container ? Normally open files can be accessed but where container are concerned there can be weird operating system stuff going on.

3

Please see watching if About → Show log → Live → Warning gets any more specific information.
Sometimes you need to click on the lines.

4

When I follow your instructions I am getting an empty page. Maybe because the last 2 days the backup was working without warning?

The warnings are quite annoying, because I think I can ignore them, but I am getting these e-mails all the time for every server i am running …

5

You mean, that at the time of the backup, some process by Prometheus is running?

6

Yes, it could be. a ‘wal’ file is evocative of an opened Sqlite database file (usually not something useful to backup in most cases)

7

OK. So what would you suggest to prevent these warnings? Is there a way to ignore these files or is it a good idea to just turn of the notifications for Warnings?

8

you can exclude paths in the UI below the directory selection window.

no, never.

9

On-disk layout supports that idea, but just like SQLite, Deleting a hot journal can lose information.
Compaction sounds like what periodically gets the raw data out of the write-ahead log files. IDK…

Possibly it’s related to compaction or some other activity done on wal files. The checkpoint folders probably won’t suffer from locking, but I’m not sure what they are, and whether they come and go.

If live log is too slow, you can let log-file=<path> using log-file-log-filter=warning monitor for awhile.
I think there’s a pull request to put a little more information in the regular log on the specific cause.

There’s generally a deeper question for databases or any complex layout that changes under you. Making the files stand still (or give a static snapshot) is necessary to get a consistent files backup.

Windows solves this with VSS, with help from applications. Prometheus looks like it can snapshot.
Prometheus TSDB (Part 2): WAL and Checkpoint and friends has some deep technical details too.

Generally one should look for some database-specific advice for backup and restore. For example:
Snapshots, backup and restore #3838 which just pointed to their mailing list – you can check there.
Feedback on time series backup strategy. How do you do it? which links to a few things people run.
Prometheus export / import data for backup and similar ideas that one can find with Internet search.
This page had one rather gloomy view of the situation. You can decide how serious you want to be.
Even a semi-serious backup deserves restore tests, so there’s something else you should consider.

10

Reading your answer leads me to the question if making backups of Prometheus in general makes sense … But I think I will just try to ignore warnings from Prometheus volumes …

11

Although it was a 2017 answer, that was the gloomy view – and accepted answer at Stack Overflow.
Choosing to not back it up will certainly give you a lot of room to exclude your way past its warnings.