Increased memory usage in 2.1.0.123

Taomyn · July 10, 2025, 7:55am

Just tried to get a bug report again, this time with .123 - got another error that mentioned disk space, but that is fine, did notice an issue with memory which I sorted, but it’s still the same:

10 Jul 2025 09:47: Failed while executing CreateReport "MAGGIE-Local_Backup" (id: 5)
Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 13: 'database or disk is full'.
   at Microsoft.Data.Sqlite.SqliteException.ThrowExceptionForRC(Int32 rc, sqlite3 db)
   at Microsoft.Data.Sqlite.SqliteDataReader.NextResult()
   at Microsoft.Data.Sqlite.SqliteCommand.ExecuteReader(CommandBehavior behavior)
   at Microsoft.Data.Sqlite.SqliteCommand.ExecuteNonQuery()
   at System.Data.Common.DbCommand.ExecuteNonQueryAsync(CancellationToken cancellationToken)
--- End of stack trace from previous location ---
   at Duplicati.Library.Main.Database.ExtensionMethods.ExecuteNonQueryAsync(SqliteCommand self, Boolean writeLog, String cmd, Dictionary`2 values, CancellationToken cancellationToken)
   at Duplicati.Library.Main.Database.ExtensionMethods.ExecuteNonQueryAsync(SqliteCommand self, String cmdtext, CancellationToken cancellationToken)
   at Duplicati.Library.Main.Database.LocalBugReportDatabase.Fix(CancellationToken token)
   at Duplicati.Library.Main.Database.LocalBugReportDatabase.Fix(CancellationToken token)
   at Duplicati.Library.Main.Operation.CreateBugReportHandler.RunAsync()
   at Duplicati.Library.Main.Operation.CreateBugReportHandler.RunAsync()
   at Duplicati.Library.Utility.Utility.Await(Task task)
   at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Func`3 method)
   at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, Func`3 method)
   at Duplicati.Library.Main.Controller.CreateLogDatabase(String targetpath)
   at Duplicati.Server.Runner.RunInternal(Connection databaseConnection, EventPollNotify eventPollNotify, INotificationUpdateService notificationUpdateService, IProgressStateProviderService progressStateProviderService, IApplicationSettings applicationSettings, IRunnerData data, Boolean fromQueue)
10 Jul 2025 09:47: Error in worker
Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 13: 'database or disk is full'.
   at Microsoft.Data.Sqlite.SqliteException.ThrowExceptionForRC(Int32 rc, sqlite3 db)
   at Microsoft.Data.Sqlite.SqliteDataReader.NextResult()
   at Microsoft.Data.Sqlite.SqliteCommand.ExecuteReader(CommandBehavior behavior)
   at Microsoft.Data.Sqlite.SqliteCommand.ExecuteNonQuery()
   at System.Data.Common.DbCommand.ExecuteNonQueryAsync(CancellationToken cancellationToken)
--- End of stack trace from previous location ---
   at Duplicati.Library.Main.Database.ExtensionMethods.ExecuteNonQueryAsync(SqliteCommand self, Boolean writeLog, String cmd, Dictionary`2 values, CancellationToken cancellationToken)
   at Duplicati.Library.Main.Database.ExtensionMethods.ExecuteNonQueryAsync(SqliteCommand self, String cmdtext, CancellationToken cancellationToken)
   at Duplicati.Library.Main.Database.LocalBugReportDatabase.Fix(CancellationToken token)
   at Duplicati.Library.Main.Database.LocalBugReportDatabase.Fix(CancellationToken token)
   at Duplicati.Library.Main.Operation.CreateBugReportHandler.RunAsync()
   at Duplicati.Library.Main.Operation.CreateBugReportHandler.RunAsync()
   at Duplicati.Library.Utility.Utility.Await(Task task)
   at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Func`3 method)
   at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, Func`3 method)
   at Duplicati.Library.Main.Controller.CreateLogDatabase(String targetpath)
   at Duplicati.Server.Runner.RunInternal(Connection databaseConnection, EventPollNotify eventPollNotify, INotificationUpdateService notificationUpdateService, IProgressStateProviderService progressStateProviderService, IApplicationSettings applicationSettings, IRunnerData data, Boolean fromQueue)
   at Duplicati.Server.Runner.Run(Connection databaseConnection, EventPollNotify eventPollNotify, INotificationUpdateService notificationUpdateService, IProgressStateProviderService progressStateProviderService, IApplicationSettings applicationSettings, IQueuedTask data, Boolean fromQueue)

A couple of extra comments:

The message that pops up showing there was an error, old an new UI, when you press the show log takes you to the job logs where this doesn’t exist as it’s in the main server logs under About.
Has .123 release (maybe the previous ones as well) started using more memory? The server was fine after upgrading to .123 - it then did a backup and I got memory alerts and saw that Duplicati process were using far more than before the backup

image1182×413 12.2 KB

image1306×497 51.9 KB

carljohnsen · July 10, 2025, 1:35pm

Hi @Taomyn!

Part of the 121+ builds that utilize the new SQLite engine has a bit more aggressive memory usage configurations by default; it has the pragmas mmap_size and cache_size both set to 64M, so I wouldn’t be too surprised if you could see up to an increase of 128M compared to previous releases. I’m not seeing the increase that you’re seeing - server process resides around ~200 MB usage. Is the numbers you’ve reported during idle? Or was it during backup / restore / etc?

The default SQLite configuration is overridden by the environment variable CUSTOMSQLITEOPTIONS (I can see that the old postfix _DUPLICATI is missing - this will be fixed in a later build once this PR has been merged). If you keep seeing the increased memory consumption, you could try to go with the old defaults, which would be

CUSTOMSQLITEOPTIONS=cache_size=-2000;mmap_size=0

for release .123.

For releases before .120 and (and presumeably .124 onwards), it would be:

CUSTOMSQLITEOPTIONS_DUPLICATI=cache_size=-2000;mmap_size=0

Taomyn · July 10, 2025, 1:54pm

I’ll see what happens on the next backup of the same server and let you know if the same happens - I won’t change anything for now, then change the setting for the next run.

Those shots were with the server at idle - it’s a file server and does nothing else, and I could not see any thing accessing it’s files.

BTW, what size databases are you testing with? On this server it has 2-3GB files, but I have another one with twice that. Just a thought in case it could be a factor.