Update: temp_store=FILE appears to be an effective workaround
I did some more testing and have a much clearer picture now.
Rolling back from 2.4.0.0 to 2.3.0.4 did not solve the problem. The same large jobs still reached the delete/cleanup phase, memory kept increasing, and duplicati-server was repeatedly killed by the Linux cgroup OOM killer.
Increasing the Docker memory limit from 3 GiB to 4 GiB and increasing host swap also did not solve it. Duplicati simply consumed more memory and eventually hit the new limit as well.
I also tried rebuilding the local database, but the rebuild itself hit the same memory problem.
I then tried the workaround mentioned in this thread:
Need recovery advice after 2.3.0.0 issue: recreate found 36 broken filelists, purge dry-run fails
In that case, a large database repair was getting stuck on a very large SQLite query over FilesetEntry / File, consuming a huge amount of memory. The workaround was:
CUSTOMSQLITEOPTIONS_DUPLICATI=temp_store=FILE
I applied the same setting and also explicitly configured a disk-backed SQLite temp directory:
CUSTOMSQLITEOPTIONS_DUPLICATI=temp_store=FILE
SQLITE_TMPDIR=/sqlite_cache
/sqlite_cache is a normal ext4 bind mount, not tmpfs.
Since this change, both of my previously failing large jobs have completed successfully:
Job 1:
source: 21.804 GiB
repository: 611.781 GiB / 189 versions
duration: 01:21:10
Job 2:
source: 18.658 GiB
repository: 737.384 GiB / 14 versions
duration: 01:05:35
One of the runs spent about 38 minutes deleting old versions, deleted 228 versions, and completed with:
Warnings: 0
Errors: 0
Before changing SQLite temp storage, I recorded 16 OOM kills of duplicati-server.
Since enabling temp_store=FILE, I have recorded 0 new OOM kills.
The memory profile also changed significantly.
Before the workaround, the kernel OOM messages showed roughly 3 GiB of anonymous RSS in duplicati-server before it was killed.
With temp_store=FILE, while a large job is running I now see something closer to:
anon : ~244 MiB
file : ~1.9 GiB
shmem: ~58 MiB
So instead of several GiB accumulating as anonymous process memory, much more of the working set is now file-backed and reclaimable by the kernel.
I also confirmed that SQLite is actually using the configured temp directory. The directory itself appears empty because SQLite opens the temp files and then unlinks them, but they are still visible through the process file descriptors, for example:
/sqlite_cache/etilqs_... (deleted) ~18 MiB
/sqlite_cache/etilqs_... (deleted) ~148 MiB
Disk I/O is clearly higher now, which is expected, but memory usage is much healthier and the large jobs are completing normally.
I cannot say whether this is exactly the same SQLite query/problem as in the linked repair thread, because my failures mainly happened during delete/cleanup and also during a database rebuild. However, the fact that the same temp_store=FILE setting changes the memory behaviour so dramatically in both cases makes this look like the same general class of large-database SQLite temp-storage problem.
For now:
CUSTOMSQLITEOPTIONS_DUPLICATI=temp_store=FILE
looks like a reliable workaround in my case.
I do not yet know how much the extra disk I/O affects total backup duration. I will keep this configuration running for the next few days and compare backup times once I have a few more normal runs.