Database or Disk Full

Hi

I have a 50GB Backup but there is a lot of file in the backup. Getting the following error

Log data:
2020-07-07 04:47:36 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.FilePreFilterProcess.FileEntry-FailedToAddFile]: Failed while attempting to add unmodified file to database:
Mono.Data.Sqlite.SqliteException (0x80004005): Insertion failed because the database is full
database or disk is full

System Centos: 7.5
Disk Has plenty of Free Space on Ext4
Version Lateste Beta 2.0.5.1_beta_2020-01-18
Mono: 6.8

Database is however close to 8GB in size.
102 Version in the backup whose size is 15GB. A lot of the data in the backup is similar small files.

Any ideas havenot tried a vaccum or repair yet. Just checkingg is there any styrange limits of the database

Eventually solved this

Quick search of the forum and reading docs SQLITE ignores the --temp-dir setting.

As I had a version of SQLITE less then 3.8.1 it also does not recognise SQLITE_TMPDIR environment variable so the disk that the temp database files where actually going to was filling up and running out of space.

As I didn’t want to change the global TMPDIR I added it as an Environment in the systemd service file for duplicat service so it only applied to this service

Issue resolved

1 Like

Glad you found a solution and thanks for sharing it!

Hi,
I have the same problem. My situation is that the global TMP is on a RAM disk which is fast (so I don’t want to change it), but it is limited in size (5GB but to small for searching all 220 versions of my backup).

What is the full path of the mentioned “systemd service file” for Duplicati service?
In another post I found /lib/systemd/system/duplicati.service which looks like a Linux path. But my Duplicati server runs on a Windows machine.

What is the corresponding file for Windows to tell Duplicati service another TMP folder?
And what exactly should I write to this file?

Anyone who can help? Thank you.

A little unclear, even with the added explanation. By global TMP, do you mean you did it like:

Windows Temporary Files – Everything you want to know by using an environment variable?

I don’t know if you have a Linux system, but looking at that would show that it says:

EnvironmentFile=-/etc/default/duplicati

which is where you can put variables for Duplicati. Windows may lack an easy way.
Harder ways seem possible using the registry. Maybe you can set this widely using:

How to see Names and Values of Environment Variables in Windows 11/10
Create Environment Variable in Windows 10
etc. and hope that your TMPDIR variable conflicts with no other program uses, BUT

tempdir

Duplicati will use the system default temporary folder. This option can be used to supply an alternative folder for temporary storage. Note that SQLite will always put temporary files in the system default temporary folder.
Consider using the TMPDIR environment variable on Linux to set the temporary folder for both Duplicati and SQLite.

which (by omission) does not sound great for Windows, if the goal is to also move SQLite files,
however Duplicati’s non-SQLite files can easily be moved just by providing its advanced option.

Temporary File Storage Locations is the SQLite manual statement on where it keeps temp files.

Do you know what you’re trying to do? Maybe you’re not that far yet, just want to stop problems. Describing the actual problem might help. There are certainly some other ways to configure this.

1 Like

If you are running Duplicati as a service on a linux system
the the /etc/systemd/system/duplicati.service file should contain EnvironmentFile location

In that file use the variable TMPDIR=/{TMP LOCATION OF CHOICE}

1 Like

Unfortunately Windows SQLite works differently, at least according to the manual linked above,
which looks even worse if you follow the links they provide. Windows SQLite looks less flexible.

Thank you ccmgr for the Linux variant of the solution. :+1:
Looks streight forward. I will keep that in mind, for the right occation.

Thank you, ts678, for the Windows version, which is not inviting as you already stated:

I am citeing from here:
Name Of The Folder Holding Temporary Files
“Applications are strongly discouraged from using this global variable. It is required to set a temporary folder on Windows Runtime (WinRT). … The temporary directory must be set prior tocalling sqlite3_open … Otherwise, various features that require theuse of temporary files may fail. … example of how to do this using C++ …”

I stop here :laughing:

The problem was, the command line call of the FIND command terminated with the error message
“code = Full (13), message = System.Data.SQLite.SQLiteException (0x800007FF): database or disk is full”,
when I made a query on a larger backup database with the –all-versions option. While running the command, I watched my TEMP folder and saw the files increasing 5GB rapidly. To overcome the error and continue my work, I relocated my TEMP folder to a larger drive (than my RAM disk is) in the Windows system dialog for changing environment variables - user variables - TEMP=D:\TEMP. In addidion I set the --tempdir option to the larger drive. These measures helped and thereby proved that the error message was correct.

I could certainly reduce my 220 versions with a proper retention policy, but will help? Because my source is 150GB and my target is 166GB which is not so bad for 220 versions in spite of compression. My strategy for the first year I have been using Duplicati was to see how the backup size would develop, and after first experience define the right retention policy to thin out the backup. But will it make a significant difference, what do you think?

Or would it be more effective to split down the source into more backup definitions?

Any more other ways to configure this?

Did names of those new files start with dup-*, or have etilqs in them?
That would help to determine what was causing fill, and how to avoid it.

While the final fail was in SQLite, something else might also use space.
dup-* files are named by Duplicati, so can be relocated using tempdir.

I had the SQLite manual suggestion in mind awhile:

The folder set by PRAGMA temp_store_directory

but then we both found that it comes with limitations.

This pragma is deprecated and exists for backwards compatibility only. New applications should avoid using this pragma. Older applications should discontinue use of this pragma at the earliest opportunity.

SQLite does what it does, unpredictably. It seems like it ought to, but testing it is more reliable.

The FIND command takes not only --all-versions but --version, whose use sometimes is

By default, Duplicati will list and restore files from the most recent backup, use this option to select another item. You may enter multiple values separated with comma, and ranges using -, e.g. 0,2-4,7.

and if that style is supported by find, you can see if this sort of trimming down will help things. Trimming down actual stored versions (as opposed to find versions) might wind up differently.

If you’ve ever successfully done --all-versions before, and everything else has been steady (hard to say, probably), then you might already have the answer of whether less versions helps.

Splitting the backup might help, but it’s probably difficult to do while keeping the older versions. Awkwardly splitting it might be possible by cloning backup, then doing different purge on each.

Deleting versions is quite easy. Fresh start is easy too, but loses all of the old backup versions.

Since this is a find, I’d guess it’s kind of name oriented. If your names or files change a lot as time passes (look at a backup log for its stats), then trimming versions will reduce total names.

Your source is slightly beyond recommended for a default 100 KB blocksize, but not enough to reduce SQLite performance greatly, and blocksize can’t be changed without fresh backup, but should you choose to do that anyway, raising blocksize will shrink databases, save space, etc., while making deduplication less effective. Your 5 GB ramdisk may be forcing some tradeoffs…

At the risk of sounding stupid,how do I go about setting the SQLITE_TMPDIR environment in systemd service file?

Why do you need it in the service file? As mentioned, service file has EnvironmentFile.
Putting environment variables in that, as separately mentioned seems like the way to go.
/etc/default/duplicati is doing that for DAEMON_OPTS variable, so that’s an example.
Environment documentation also covers direct environment changes, if you insist on that.

1 Like

Thanks! I will do that