How to backup Checkmk running in Docker

tobias.graf · December 18, 2023, 4:25pm

Hi

I run Checkmk Raw in a Docker container and try to backup it using Duplicati. What is the best way to backup Checkmk in this scenario?

Currently I am mounting the /omd/sites directory into the container using the -v /data/cmk/monitoring:/omd/sites parameter as documented here: Installation als Docker-Container
I use Duplicati to backup the data at /data/cmk/monitoring on a daily basis. I assume this is basically enough to backup Checkmk running in Docker since I can stop/remove the container and start it again and everything is still in place. However sometimes, I get such kind of errors from Duplicati when it tries to backup the files (while Checkmk is also running of course):

2023-12-12 00:00:08 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.MetadataGenerator.Metadata-MetadataProcessFailed]: Failed to process metadata for "/host/mnt/volume1/data/docker/volumes/check-mk-raw/omd/sites/cmk/var/pnp4nagios/spool/perfdata.1702335593", storing empty metadata
UnixSupport.File+FileAccesException: Unable to access the file "/host/mnt/volume1/data/docker/volumes/check-mk-raw/omd/sites/cmk/var/pnp4nagios/spool/perfdata.1702335593" with method llistxattr, error: ENOENT (2)
  at UnixSupport.File.GetExtendedAttributes (System.String path, System.Boolean isSymlink, System.Boolean followSymlink) [0x00087] in <3a2c307381104a73ba105ad3712d666f>:0
  at Duplicati.Library.Common.IO.SystemIOLinux.GetMetadata (System.String file, System.Boolean isSymlink, System.Boolean followSymlink) [0x0000d] in <6e4f34e913fd4d89b628a5a76045f19c>:0
  at Duplicati.Library.Snapshots.NoSnapshotLinux.GetMetadata (System.String localPath, System.Boolean isSymlink, System.Boolean followSymlink) [0x00000] in <acb4fdca179441ff87fffb9a1b9a62c8>:0
  at Duplicati.Library.Main.Operation.Backup.MetadataGenerator.GenerateMetadata (System.String path, System.IO.FileAttributes attributes, Duplicati.Library.Main.Options options, Duplicati.Library.Snapshots.ISnapshotService snapshot) [0x0001b] in <e60bc008dd1b454d861cfacbdd3760b9>:0
2023-12-12 00:00:08 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: /host/mnt/volume1/data/docker/volumes/check-mk-raw/omd/sites/cmk/var/pnp4nagios/spool/perfdata.1702335593
System.IO.FileNotFoundException: Could not find file "/host/mnt/volume1/data/docker/volumes/check-mk-raw/omd/sites/cmk/var/pnp4nagios/spool/perfdata.1702335593"
File name: '/host/mnt/volume1/data/docker/volumes/check-mk-raw/omd/sites/cmk/var/pnp4nagios/spool/perfdata.1702335593'
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.Boolean anonymous, System.IO.FileOptions options) [0x0019e] in <d13c8b563008422a8c5aaec0a74089cc>:0
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share) [0x00000] in <d13c8b563008422a8c5aaec0a74089cc>:0
  at (wrapper remoting-invoke-with-check) System.IO.FileStream..ctor(string,System.IO.FileMode,System.IO.FileAccess,System.IO.FileShare)
  at System.IO.File.Open (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share) [0x00000] in <d13c8b563008422a8c5aaec0a74089cc>:0
  at Duplicati.Library.Snapshots.SnapshotBase.OpenRead (System.String localPath) [0x00000] in <acb4fdca179441ff87fffb9a1b9a62c8>:0
  at Duplicati.Library.Main.Operation.Backup.FileBlockProcessor+<>c__DisplayClass1_0.<Run>b__0 (<>f__AnonymousType8`2[<Input>j__TPar,<StreamBlockChannel>j__TPar] self) [0x00217] in <e60bc008dd1b454d861cfacbdd3760b9>:0

Seems like some files are in use or not available anymore when Duplicati tries to backup them.

I am thinking of the following options:

Should I exclude some directories/files from the Duplicati backup job?
Should I play around with options like this in Duplicati? Advanced Options - Duplicati 2 User's Manual
Should I rather use the built-in backup/restore feature of Checkmk as described here? Backups - Daten sichern und wiederherstellen

How can I address such situations in Duplicati in general?

Any advice is appreciated.

ts678 · December 18, 2023, 5:50pm

When talking Docker, paths are often confusing (e.g. is it inside or outside the container), but that isn’t what’s shown in the error messages you quoted next. Is Duplicati in a container as well? What image?

Installation with Docker from earlier today has some notes and links to the general Docker information. Specifics of this system may matter, and for checkmk itself, experts are at checkmk.com and its forum:

might be better answered there. Google saw quite a bit of backup, Docker, and container commentary.

Are you able to stop it? Backups of things that stay running risk files disappearing under them, maybe

System.IO.FileNotFoundException: Could not find file "/host/mnt/volume1/data/docker/volumes/check-mk-raw/omd/sites/cmk/var/pnp4nagios/spool/perfdata.1702335593"

(and you can see the difference between that and /data/cmk/monitoring backup, so what’s where?)

With complicated systems with lots of moving parts, the plan is sometimes to take a snapshot (so it sits still while being backed up), and there are careful directions for how to restore (don’t forget that part…).

Facilities for this are provided by the application. Databases are often this way. A live backup may grab something, but it won’t be internally consistent, so might not restore and run. See checkmk information.

When asking clarifying questions, you might not want to specifically say Duplicati, but Docker might be relevant. You need advice for a running (or does that matter?) checkmk for file oriented backup/restore.

drwtsn32 · December 18, 2023, 10:32pm

IMO data you want to keep should be saved outside the docker container. This allows you to reset/redeploy docker containers without data loss. I have had no issues using Duplicati to back up data saved outside of containers like this. (I’ve never tried using it to back up the containers themselves, but there’s no reason for me to do that when I don’t store important data inside the container.)

ts678 · December 18, 2023, 11:52pm

Backup docker container, databases, volumes in the Docker forum agrees, and says more.

As for the current topic, I’m not sure where Duplicati is, and what it’s configured to back up.
Possibly it’s in its own Docker, attempting to backup checkmk which is in a different Docker.

OP might be asking the container backup question above, where answer was don’t do that.
Docker itself is a complex application, so attempting to grab then restore the pieces is hard.

Easier and simpler is to use checkmk facilities to snapshot state to host, then back state up.
That will prevent the backup from backing up partial files or having intended files disappear.

Container should be disposable. I know a test was done but it wasn’t described very clearly,
or possibly I’m not Docker-savvy enough to follow what it was, and what exactly that proves.