Strategy to recover from RAM corrupted blocks

obones · June 10, 2025, 6:56pm

Ok, I tried for both my backups where “repair” fail with the same error and cannot get the create-report operation to work as I get those errors:

The operation CreateLogDb has failed => No space left on device

System.IO.IOException: No space left on device
   at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirError)
   at System.IO.FileSystem.CopyFile(String sourceFullPath, String destFullPath, Boolean overwrite)
   at Duplicati.Library.Main.Operation.CreateBugReportHandler.RunAsync()
   at Duplicati.Library.Utility.Utility.Await(Task task)
   at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Func`3 method)
   at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, Func`3 method)
   at Duplicati.Library.Main.Controller.CreateLogDatabase(String targetpath)
   at Duplicati.CommandLine.Commands.CreateBugReport(TextWriter outwriter, Action`1 setup, List`1 args, Dictionary`2 options, IFilter filter)
   at Duplicati.CommandLine.Program.ParseCommandLine(TextWriter outwriter, Action`1 setup, Boolean& verboseErrors, String[] args)
   at Duplicati.CommandLine.Program.RunCommandLine(TextWriter outwriter, TextWriter errwriter, Action`1 setup, String[] args)

That’s with the larger database, about 12G in size. As it doesn’t tell me which device is missing space, I have no idea where to look, especially because the system has quite a bit of free space:

Sys. de fichiers Taille Utilisé Dispo Uti% Monté sur
/dev/sdb2          424G    229G  174G  57% /
tmpfs               16G    3,3M   16G   1% /dev/shm
tmpfs               16G     12G  4,1G  75% /tmp
/dev/sdb1          300M    312K  300M   1% /boot/efi
/dev/sdd1          1,8T    993G  748G  58% /mnt/backup
tmpfs              3,2G    124K  3,2G   1% /run/user/1000
/dev/sde1          2,7T    2,0T  611G  77% /mnt/bay

The source files are inside /mnt/backup, the database in /var/lib/duplicati/.config/Duplicati/ and the remote content has been stored locally inside /mnt/bay

I tried with the smaller database, about 2.5G in size and this time I get this:

The operation CreateLogDb has failed => database or disk is full
database or disk is full

code = Full (13), message = System.Data.SQLite.SQLiteException (0x800007FF): database or disk is full
database or disk is full
   at System.Data.SQLite.SQLite3.Reset(SQLiteStatement stmt)
   at System.Data.SQLite.SQLite3.Step(SQLiteStatement stmt)
   at System.Data.SQLite.SQLiteDataReader.NextResult()
   at System.Data.SQLite.SQLiteDataReader..ctor(SQLiteCommand cmd, CommandBehavior behave)
   at System.Data.SQLite.SQLiteCommand.ExecuteReader(CommandBehavior behavior)
   at System.Data.SQLite.SQLiteCommand.ExecuteNonQuery(CommandBehavior behavior)
   at System.Data.SQLite.SQLiteCommand.ExecuteNonQuery()
   at Duplicati.Library.Main.Database.ExtensionMethods.ExecuteNonQuery(IDbCommand self, Boolean writeLog, String cmd)
   at Duplicati.Library.Main.Database.ExtensionMethods.ExecuteNonQuery(IDbCommand self, String cmd)
   at Duplicati.Library.Main.Database.LocalBugReportDatabase.Fix()
   at Duplicati.Library.Main.Operation.CreateBugReportHandler.RunAsync()
   at Duplicati.Library.Utility.Utility.Await(Task task)
   at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Func`3 method)
   at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, Func`3 method)
   at Duplicati.Library.Main.Controller.CreateLogDatabase(String targetpath)
   at Duplicati.CommandLine.Commands.CreateBugReport(TextWriter outwriter, Action`1 setup, List`1 args, Dictionary`2 options, IFilter filter)
   at Duplicati.CommandLine.Program.ParseCommandLine(TextWriter outwriter, Action`1 setup, Boolean& verboseErrors, String[] args)
   at Duplicati.CommandLine.Program.RunCommandLine(TextWriter outwriter, TextWriter errwriter, Action`1 setup, String[] args)

I have no idea what it’s trying to do here, and why it would need so much free space that it consumes what’s available.

ts678 · June 10, 2025, 7:32pm

Presumably because SQLite didn’t say.

Probably this SQLite operation, which helps improve privacy by removing free space.
It also reduces the size of the database:

Transient Database Used By VACUUM

The VACUUM command works by creating a temporary file and then rebuilding the entire database into that temporary file. Then the content of the temporary file is copied back into the original database file and the temporary file is deleted.

3. How VACUUM works explains a third use of space for the journal, but that would be alongside the database location normally. In this case, DB is possibly is also a tmp file. Seeing even your 2.5 GB database having trouble suggest you might need the 2.5 GB database and then three times the DB space in /tmp, so trying to fit 7.5 GB into 4.1 GB.

Regardless, unless a more authoritative answer comes, maybe you can relocate temp:

  --tempdir (Path): Temporary storage folder
    Use this option to supply an alternative folder for temporary storage. By default
    the system default temporary folder is used. Note that also SQLite will put
    temporary files in this temporary folder.

This is the new text to go with new code. The old text was gloomier about ability to move temporary storage on Linux, so I hope the new text is accurate in what it says for SQLite.

EDIT 1:

github.com/duplicati/duplicati

Duplicati/Library/Main/Operation/CreateBugReportHandler.cs

8797d3130


      
          using (var tmp = new TempFile())

          {

              File.Copy(m_options.Dbpath, tmp, true);

              using (var db = new LocalBugReportDatabase(tmp, m_options.SqlitePageCache))

              {

                  db.Fix();

                  if (m_options.AutoVacuum)

                      db.Vacuum();

              }

That should force at least one DB into the relocated tmp, but I’m not sure it can force all.
5. Temporary File Storage Locations gives me some hope that Duplicati can persuade it.

kenkendk · June 12, 2025, 4:04pm

Usually, this is the TMP or TMPDIR variable that defines where the temporary file goes. I would guess /tmp is where it tries to place it.

ts678 · June 12, 2025, 6:08pm

Does Duplicati follow those? At least per the help text, tempdir option seems completer.
For example, TempFile has to point to somewhere bigger than 4.1 GB to take 12 GB DB which is just the starting point before SQLite vacuum happens to further clean up that DB.

obones · June 13, 2025, 7:11am

The --tempdir option worked fine, I was able to generate the bug report and have sent a PM to @kenkendk, thanks for pointing it out.

kenkendk · June 14, 2025, 9:57am

Duplicati’s default for --tempdir is the “user temporary folder”, which on Linux is usually derived from the TMP / TMPDIR environment variable.

Duplicati tries very hard to force SQLite to use whatever path is set with --tempdir, but SQLite has deprecated all methods to consistently choose a temporary folder, in favor of just using the system default.

Thanks!

ts678 · June 14, 2025, 12:20pm

It looks like it uses System.IO.Path.GetTempPath which is $TMPDIR else just /tmp, basically being both responsive to incoming setting, and able to override that default.

I can agree with that more on Windows. On UNIX-like systems, it’s extremely defined:

Gives UNIX-like systems a huge number of ways to set the temporary folder. Windows doesn’t offer quite as much direct control, but goes through a Windows call which does.

Path.GetTempPath Method again, but on Windows tab which I think cites functions like:

GetTempPath2W function (fileapi.h)

GetTempPathW function (fileapi.h)

both of which have a long list (including environment variables) that can move the folder.
Or at least that’s how it looks to me.

kenkendk · June 16, 2025, 2:36pm

From the smaller bugreport database, I have identified the issue.

The problem is that two directories are missing metadata. Somehow, these two entries have been stored with metadata that has length zero. Because a length of zero means that there is no data, this metadata points to a blockset that has length zero, and then no blocks.

This trips up the validation, because there are now two entries that are not correctly represented in the database.

I am sure that Duplicati has somehow mangaged to create these empty entries, but so far I have not been able to figure out the scenario that causes it. I am leaning towards this being a result of the defective RAM blocks, but I would prefer something more solid.

For the fix, I am implementing an update to the repair process that detects these entries and replaces them with different metadata. This process is also used during purge-broken-files, where it opts to recover data by using incorrect metadata instead of simply deleting files that are missing metadata.

The problem is one step deeper:

Pragma statements supported by SQLite

This pragma is deprecated and exists for backwards compatibility only. New applications should avoid using this pragma. Older applications should discontinue use of this pragma at the earliest opportunity. This pragma may be omitted from the build when SQLite is compiled using SQLITE_OMIT_DEPRECATED.

ts678 · June 16, 2025, 3:05pm

Then don’t. The next step goes to Windows, which follows the environment variables, etc.
Quoting the warning that search option 1 leads to is good, but what’s wrong with option 2?

On Windows systems, folders are searched in the following order:

The folder set by PRAGMA temp_store_directory or by the sqlite3_temp_directory global variable

The folder returned by the GetTempPath() system interface.

SQLite itself does not pay any attention to environment variables in this case, though presumably the GetTempPath() system call does

I cited documentation from Microsoft on their GetTempPath* family already. Any issues?

EDIT 1:

The sqlite3_temp_directory handling is unknown, so I think I’m going to have to test.

EDIT 2:

Testing --tempdir on 2.1.0.119, the one file in usual Temp was a dupl-usagereport-*.

EDIT 3:

Removed --tempdir to test that default setting of that comes through Windows settings.

set on my Command Prompt has both TMP and TEMP set to the usual spot, but YMMV.

Set TMP environment somewhere else before TrayIcon start, and nothing is at usual spot.
Same result if I set TMP= to delete it, and then set TEMP elsewhere. All seems per docs, however there are other ways, and also SYSTEM may be different to improve the security.

The path specified by the TMP environment variable.

The path specified by the TEMP environment variable.

kenkendk · June 17, 2025, 7:46am

The problem with environment variables is that it is not clear when they are read.
For most applications, it is fine: read the temp folder on startup, then carry on.

For Duplicati, it should be possible to set a different temp folder for each backup job.
If we start fiddling with the environment variables, we may cause changes in the server as well (it is the same process). If the SQLite library caches this variable on load, it will not be possible to change it for later runs.

This will of course be fixed if we run each backup task as a separate process, but that is a non-trivial change.