Strategy to recover from RAM corrupted blocks

Support
21

Ok, I tried for both my backups where “repair” fail with the same error and cannot get the create-report operation to work as I get those errors:

The operation CreateLogDb has failed => No space left on device

System.IO.IOException: No space left on device
   at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirError)
   at System.IO.FileSystem.CopyFile(String sourceFullPath, String destFullPath, Boolean overwrite)
   at Duplicati.Library.Main.Operation.CreateBugReportHandler.RunAsync()
   at Duplicati.Library.Utility.Utility.Await(Task task)
   at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Func`3 method)
   at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, Func`3 method)
   at Duplicati.Library.Main.Controller.CreateLogDatabase(String targetpath)
   at Duplicati.CommandLine.Commands.CreateBugReport(TextWriter outwriter, Action`1 setup, List`1 args, Dictionary`2 options, IFilter filter)
   at Duplicati.CommandLine.Program.ParseCommandLine(TextWriter outwriter, Action`1 setup, Boolean& verboseErrors, String[] args)
   at Duplicati.CommandLine.Program.RunCommandLine(TextWriter outwriter, TextWriter errwriter, Action`1 setup, String[] args)

That’s with the larger database, about 12G in size. As it doesn’t tell me which device is missing space, I have no idea where to look, especially because the system has quite a bit of free space:

Sys. de fichiers Taille Utilisé Dispo Uti% Monté sur
/dev/sdb2          424G    229G  174G  57% /
tmpfs               16G    3,3M   16G   1% /dev/shm
tmpfs               16G     12G  4,1G  75% /tmp
/dev/sdb1          300M    312K  300M   1% /boot/efi
/dev/sdd1          1,8T    993G  748G  58% /mnt/backup
tmpfs              3,2G    124K  3,2G   1% /run/user/1000
/dev/sde1          2,7T    2,0T  611G  77% /mnt/bay

The source files are inside /mnt/backup, the database in /var/lib/duplicati/.config/Duplicati/ and the remote content has been stored locally inside /mnt/bay

I tried with the smaller database, about 2.5G in size and this time I get this:

The operation CreateLogDb has failed => database or disk is full
database or disk is full

code = Full (13), message = System.Data.SQLite.SQLiteException (0x800007FF): database or disk is full
database or disk is full
   at System.Data.SQLite.SQLite3.Reset(SQLiteStatement stmt)
   at System.Data.SQLite.SQLite3.Step(SQLiteStatement stmt)
   at System.Data.SQLite.SQLiteDataReader.NextResult()
   at System.Data.SQLite.SQLiteDataReader..ctor(SQLiteCommand cmd, CommandBehavior behave)
   at System.Data.SQLite.SQLiteCommand.ExecuteReader(CommandBehavior behavior)
   at System.Data.SQLite.SQLiteCommand.ExecuteNonQuery(CommandBehavior behavior)
   at System.Data.SQLite.SQLiteCommand.ExecuteNonQuery()
   at Duplicati.Library.Main.Database.ExtensionMethods.ExecuteNonQuery(IDbCommand self, Boolean writeLog, String cmd)
   at Duplicati.Library.Main.Database.ExtensionMethods.ExecuteNonQuery(IDbCommand self, String cmd)
   at Duplicati.Library.Main.Database.LocalBugReportDatabase.Fix()
   at Duplicati.Library.Main.Operation.CreateBugReportHandler.RunAsync()
   at Duplicati.Library.Utility.Utility.Await(Task task)
   at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Func`3 method)
   at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, Func`3 method)
   at Duplicati.Library.Main.Controller.CreateLogDatabase(String targetpath)
   at Duplicati.CommandLine.Commands.CreateBugReport(TextWriter outwriter, Action`1 setup, List`1 args, Dictionary`2 options, IFilter filter)
   at Duplicati.CommandLine.Program.ParseCommandLine(TextWriter outwriter, Action`1 setup, Boolean& verboseErrors, String[] args)
   at Duplicati.CommandLine.Program.RunCommandLine(TextWriter outwriter, TextWriter errwriter, Action`1 setup, String[] args)

I have no idea what it’s trying to do here, and why it would need so much free space that it consumes what’s available.

22

Presumably because SQLite didn’t say.

Probably this SQLite operation, which helps improve privacy by removing free space.
It also reduces the size of the database:

Transient Database Used By VACUUM

The VACUUM command works by creating a temporary file and then rebuilding the entire database into that temporary file. Then the content of the temporary file is copied back into the original database file and the temporary file is deleted.

3. How VACUUM works explains a third use of space for the journal, but that would be alongside the database location normally. In this case, DB is possibly is also a tmp file. Seeing even your 2.5 GB database having trouble suggest you might need the 2.5 GB database and then three times the DB space in /tmp, so trying to fit 7.5 GB into 4.1 GB.

Regardless, unless a more authoritative answer comes, maybe you can relocate temp:

  --tempdir (Path): Temporary storage folder
    Use this option to supply an alternative folder for temporary storage. By default
    the system default temporary folder is used. Note that also SQLite will put
    temporary files in this temporary folder.

This is the new text to go with new code. The old text was gloomier about ability to move temporary storage on Linux, so I hope the new text is accurate in what it says for SQLite.

EDIT 1:

That should force at least one DB into the relocated tmp, but I’m not sure it can force all.
5. Temporary File Storage Locations gives me some hope that Duplicati can persuade it.

23

Usually, this is the TMP or TMPDIR variable that defines where the temporary file goes. I would guess /tmp is where it tries to place it.

24

Does Duplicati follow those? At least per the help text, tempdir option seems completer.
For example, TempFile has to point to somewhere bigger than 4.1 GB to take 12 GB DB which is just the starting point before SQLite vacuum happens to further clean up that DB.

25

The --tempdir option worked fine, I was able to generate the bug report and have sent a PM to @kenkendk, thanks for pointing it out.

1 Like
26

Duplicati’s default for --tempdir is the “user temporary folder”, which on Linux is usually derived from the TMP / TMPDIR environment variable.

Duplicati tries very hard to force SQLite to use whatever path is set with --tempdir, but SQLite has deprecated all methods to consistently choose a temporary folder, in favor of just using the system default.

Thanks!

27

It looks like it uses System.IO.Path.GetTempPath which is $TMPDIR else just /tmp, basically being both responsive to incoming setting, and able to override that default.

I can agree with that more on Windows. On UNIX-like systems, it’s extremely defined:

Gives UNIX-like systems a huge number of ways to set the temporary folder. Windows doesn’t offer quite as much direct control, but goes through a Windows call which does.

Path.GetTempPath Method again, but on Windows tab which I think cites functions like:

GetTempPath2W function (fileapi.h)

GetTempPathW function (fileapi.h)

both of which have a long list (including environment variables) that can move the folder.
Or at least that’s how it looks to me.