# Decline in performance after update to 2.1.0.2\_beta? **URL:** \ **Category:** Support\ **Created:** [January 27, 2025, 4:08pm UTC](https://forum.duplicati.com/t/decline-in-performance-after-update-to-2-1-0-2-beta/20024 "2025-01-27T16:08:25Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![redusr](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/r/bcef8e/32.png) [@redusr](https://forum.duplicati.com/u/redusr)\ **Post date:** [January 27, 2025, 4:08pm UTC](https://forum.duplicati.com/t/decline-in-performance-after-update-to-2-1-0-2-beta/20024/1 "2025-01-27T16:08:25Z")
There seems to be a performance issue when running my jobs after migration from 2.0.8.1 to 2.1.0.2 - my two backup jobs appear to take up to 4x more time than before: The mean duration with 2.0.8.1 was 30-35 min (5-10 min if run a second time directly afterwards), with 2.1.0.2 the duration is about 120 min (~30-90 min the second time). The backup destination path for each job is on a separate, physically connected USB drive. Both jobs have about 10 backup versions on the remote. The behavior of increased job time applies to both jobs. When noticing the increased processing time directly after upgrading I ignored it first because I assumed that the job database (and maybe also the remote data) may have to undergo a migration procedure once that takes some additional time. Now that the jobs have run some more times with comparable processing times, I conclude that the increase in processing time must come from somewhere else. Example for job run on old duplicati (last job run on version 2.0.8.1): > Begin 2024-12-16 11:49:54 > End 2024-12-16 12:23:24 > Duration 00:33:30 ```plaintext { "DeletedFiles": 11904, "DeletedFolders": 366, "ModifiedFiles": 3524, "ExaminedFiles": 2249913, "OpenedFiles": 20690, "AddedFiles": 17166, "SizeOfModifiedFiles": 32269351944, "SizeOfAddedFiles": 14798875303, "SizeOfExaminedFiles": 305694366848, "SizeOfOpenedFiles": 47083894936, "NotProcessedFiles": 0, "AddedFolders": 629, "TooLargeFiles": 0, "FilesWithError": 0, "ModifiedFolders": 0, "ModifiedSymlinks": 0, "AddedSymlinks": 0, "DeletedSymlinks": 0, "PartialBackup": false, "Dryrun": false, "MainOperation": "Backup", "CompactResults": { "DeletedFileCount": 4, "DownloadedFileCount": 0, "UploadedFileCount": 0, "DeletedFileSize": 209350164, "DownloadedFileSize": 0, "UploadedFileSize": 0, "Dryrun": false, "VacuumResults": null, "MainOperation": "Compact", "ParsedResult": "Success", "Interrupted": false, "Version": "2.0.8.1 (2.0.8.1_beta_2024-05-07)", "EndTime": "2024-12-16T11:23:20.4952309Z", "BeginTime": "2024-12-16T11:22:11.9930518Z", "Duration": "00:01:08.5021791", "MessagesActualLength": 0, "WarningsActualLength": 0, "ErrorsActualLength": 0, "Messages": null, "Warnings": null, "Errors": null, "BackendStatistics": { "RemoteCalls": 51, "BytesUploaded": 2231243765, "BytesDownloaded": 279669335, "FilesUploaded": 41, "FilesDownloaded": 3, "FilesDeleted": 5, "FoldersCreated": 0, "RetryAttempts": 0, "UnknownFileSize": 0, "UnknownFileCount": 0, "KnownFileCount": 3505, "KnownFileSize": 183833651725, "LastBackupDate": "2024-12-16T11:49:54+01:00", "BackupListCount": 10, "TotalQuotaSpace": 536870645760, "FreeQuotaSpace": 262983827456, "AssignedQuotaSpace": -1, "ReportedQuotaError": false, "ReportedQuotaWarning": false, "MainOperation": "Backup", "ParsedResult": "Success", "Interrupted": false, "Version": "2.0.8.1 (2.0.8.1_beta_2024-05-07)", "EndTime": "0001-01-01T00:00:00", "BeginTime": "2024-12-16T10:49:54.0957413Z", "Duration": "00:00:00", "MessagesActualLength": 0, "WarningsActualLength": 0, "ErrorsActualLength": 0, "Messages": null, "Warnings": null, "Errors": null } }, "VacuumResults": null, "DeleteResults": { "DeletedSetsActualLength": 1, "DeletedSets": [ { "Item1": 3, "Item2": "2024-11-05T16:32:37+01:00" } ], "Dryrun": false, "MainOperation": "Delete", "CompactResults": { "DeletedFileCount": 4, "DownloadedFileCount": 0, "UploadedFileCount": 0, "DeletedFileSize": 209350164, "DownloadedFileSize": 0, "UploadedFileSize": 0, "Dryrun": false, "VacuumResults": null, "MainOperation": "Compact", "ParsedResult": "Success", "Interrupted": false, "Version": "2.0.8.1 (2.0.8.1_beta_2024-05-07)", "EndTime": "2024-12-16T11:23:20.4952309Z", "BeginTime": "2024-12-16T11:22:11.9930518Z", "Duration": "00:01:08.5021791", "MessagesActualLength": 0, "WarningsActualLength": 0, "ErrorsActualLength": 0, "Messages": null, "Warnings": null, "Errors": null, "BackendStatistics": { "RemoteCalls": 51, "BytesUploaded": 2231243765, "BytesDownloaded": 279669335, "FilesUploaded": 41, "FilesDownloaded": 3, "FilesDeleted": 5, "FoldersCreated": 0, "RetryAttempts": 0, "UnknownFileSize": 0, "UnknownFileCount": 0, "KnownFileCount": 3505, "KnownFileSize": 183833651725, "LastBackupDate": "2024-12-16T11:49:54+01:00", "BackupListCount": 10, "TotalQuotaSpace": 536870645760, "FreeQuotaSpace": 262983827456, "AssignedQuotaSpace": -1, "ReportedQuotaError": false, "ReportedQuotaWarning": false, "MainOperation": "Backup", "ParsedResult": "Success", "Interrupted": false, "Version": "2.0.8.1 (2.0.8.1_beta_2024-05-07)", "EndTime": "0001-01-01T00:00:00", "BeginTime": "2024-12-16T10:49:54.0957413Z", "Duration": "00:00:00", "MessagesActualLength": 0, "WarningsActualLength": 0, "ErrorsActualLength": 0, "Messages": null, "Warnings": null, "Errors": null } }, "ParsedResult": "Success", "Interrupted": false, "Version": "2.0.8.1 (2.0.8.1_beta_2024-05-07)", "EndTime": "2024-12-16T11:23:20.4952309Z", "BeginTime": "2024-12-16T11:21:24.9352481Z", "Duration": "00:01:55.5599828", "MessagesActualLength": 0, "WarningsActualLength": 0, "ErrorsActualLength": 0, "Messages": null, "Warnings": null, "Errors": null, "BackendStatistics": { "RemoteCalls": 51, "BytesUploaded": 2231243765, "BytesDownloaded": 279669335, "FilesUploaded": 41, "FilesDownloaded": 3, "FilesDeleted": 5, "FoldersCreated": 0, "RetryAttempts": 0, "UnknownFileSize": 0, "UnknownFileCount": 0, "KnownFileCount": 3505, "KnownFileSize": 183833651725, "LastBackupDate": "2024-12-16T11:49:54+01:00", "BackupListCount": 10, "TotalQuotaSpace": 536870645760, "FreeQuotaSpace": 262983827456, "AssignedQuotaSpace": -1, "ReportedQuotaError": false, "ReportedQuotaWarning": false, "MainOperation": "Backup", "ParsedResult": "Success", "Interrupted": false, "Version": "2.0.8.1 (2.0.8.1_beta_2024-05-07)", "EndTime": "0001-01-01T00:00:00", "BeginTime": "2024-12-16T10:49:54.0957413Z", "Duration": "00:00:00", "MessagesActualLength": 0, "WarningsActualLength": 0, "ErrorsActualLength": 0, "Messages": null, "Warnings": null, "Errors": null } }, "RepairResults": null, "TestResults": { "MainOperation": "Test", "VerificationsActualLength": 3, "Verifications": [ { "Key": "duplicati-20241216T104954Z.dlist.zip.aes", "Value": [] }, { "Key": "duplicati-i47ed0a26b763488bb49284d22b527f35.dindex.zip.aes", "Value": [] }, { "Key": "duplicati-b9bd605a30eae4dbeb2d508fb2508bc70.dblock.zip.aes", "Value": [] } ], "ParsedResult": "Success", "Interrupted": false, "Version": "2.0.8.1 (2.0.8.1_beta_2024-05-07)", "EndTime": "2024-12-16T11:23:24.559538Z", "BeginTime": "2024-12-16T11:23:23.1908276Z", "Duration": "00:00:01.3687104", "MessagesActualLength": 0, "WarningsActualLength": 0, "ErrorsActualLength": 0, "Messages": null, "Warnings": null, "Errors": null, "BackendStatistics": { "RemoteCalls": 51, "BytesUploaded": 2231243765, "BytesDownloaded": 279669335, "FilesUploaded": 41, "FilesDownloaded": 3, "FilesDeleted": 5, "FoldersCreated": 0, "RetryAttempts": 0, "UnknownFileSize": 0, "UnknownFileCount": 0, "KnownFileCount": 3505, "KnownFileSize": 183833651725, "LastBackupDate": "2024-12-16T11:49:54+01:00", "BackupListCount": 10, "TotalQuotaSpace": 536870645760, "FreeQuotaSpace": 262983827456, "AssignedQuotaSpace": -1, "ReportedQuotaError": false, "ReportedQuotaWarning": false, "MainOperation": "Backup", "ParsedResult": "Success", "Interrupted": false, "Version": "2.0.8.1 (2.0.8.1_beta_2024-05-07)", "EndTime": "0001-01-01T00:00:00", "BeginTime": "2024-12-16T10:49:54.0957413Z", "Duration": "00:00:00", "MessagesActualLength": 0, "WarningsActualLength": 0, "ErrorsActualLength": 0, "Messages": null, "Warnings": null, "Errors": null } }, "ParsedResult": "Success", "Interrupted": false, "Version": "2.0.8.1 (2.0.8.1_beta_2024-05-07)", "EndTime": "2024-12-16T11:23:24.5785707Z", "BeginTime": "2024-12-16T10:49:54.0957413Z", "Duration": "00:33:30.4828294", "MessagesActualLength": 116, "WarningsActualLength": 0, "ErrorsActualLength": 0, "Messages": [ "2024-12-16 11:49:54 +01 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started", "2024-12-16 11:53:02 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: ()", "2024-12-16 11:53:06 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: (3.39 KB)", "2024-12-16 11:53:06 +01 - [Information-Duplicati.Library.Main.Operation.BackupHandler-SkipUsnForVolume]: Performing full scan for volume \"C:\\\": USN journal entries were purged since last scan", "2024-12-16 12:15:48 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b065cec82db3a47d384a7418306adbd72.dblock.zip.aes (99.78 MB)", "2024-12-16 12:15:48 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-bfd37d43d11c241eca04d2830e7e4e292.dblock.zip.aes (99.79 MB)", "2024-12-16 12:15:48 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-bec570a7ddea248338a14bb800aa8e097.dblock.zip.aes (99.79 MB)", "2024-12-16 12:15:48 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-bdf3105f353d346878b743c2438b7f44d.dblock.zip.aes (99.79 MB)", "2024-12-16 12:15:56 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-bdf3105f353d346878b743c2438b7f44d.dblock.zip.aes (99.79 MB)", "2024-12-16 12:15:56 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b065cec82db3a47d384a7418306adbd72.dblock.zip.aes (99.78 MB)", "2024-12-16 12:15:56 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-bec570a7ddea248338a14bb800aa8e097.dblock.zip.aes (99.79 MB)", "2024-12-16 12:15:56 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-bfd37d43d11c241eca04d2830e7e4e292.dblock.zip.aes (99.79 MB)", "2024-12-16 12:15:56 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-if543d17afea14a53abcca27977e81269.dindex.zip.aes (205.76 KB)", "2024-12-16 12:15:56 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-i9a87da2802c345fbafdc3072df2c06bc.dindex.zip.aes (202.22 KB)", "2024-12-16 12:15:56 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-i4f95b37dba184cae96283778d38db341.dindex.zip.aes (208.22 KB)", "2024-12-16 12:15:56 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-i6d345473f0a74320b2e8cd192fa8ab31.dindex.zip.aes (194.00 KB)", "2024-12-16 12:15:56 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-if543d17afea14a53abcca27977e81269.dindex.zip.aes (205.76 KB)", "2024-12-16 12:15:56 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-bf3a046a25ef349c2a718e8a9b1aa8e42.dblock.zip.aes (99.82 MB)", "2024-12-16 12:15:57 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-i4f95b37dba184cae96283778d38db341.dindex.zip.aes (208.22 KB)", "2024-12-16 12:15:57 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b64a1261f418b4dfaa3e72f5147aa8264.dblock.zip.aes (99.77 MB)" ], "Warnings": [], "Errors": [], "BackendStatistics": { "RemoteCalls": 51, "BytesUploaded": 2231243765, "BytesDownloaded": 279669335, "FilesUploaded": 41, "FilesDownloaded": 3, "FilesDeleted": 5, "FoldersCreated": 0, "RetryAttempts": 0, "UnknownFileSize": 0, "UnknownFileCount": 0, "KnownFileCount": 3505, "KnownFileSize": 183833651725, "LastBackupDate": "2024-12-16T11:49:54+01:00", "BackupListCount": 10, "TotalQuotaSpace": 536870645760, "FreeQuotaSpace": 262983827456, "AssignedQuotaSpace": -1, "ReportedQuotaError": false, "ReportedQuotaWarning": false, "MainOperation": "Backup", "ParsedResult": "Success", "Interrupted": false, "Version": "2.0.8.1 (2.0.8.1_beta_2024-05-07)", "EndTime": "0001-01-01T00:00:00", "BeginTime": "2024-12-16T10:49:54.0957413Z", "Duration": "00:00:00", "MessagesActualLength": 0, "WarningsActualLength": 0, "ErrorsActualLength": 0, "Messages": null, "Warnings": null, "Errors": null } } ``` Example for job run on duplicati 2.1.0.2 (second job run on that release): > Begin 2025-01-14 17:32:32 > End 2025-01-14 19:03:26 > Duration 01:30:55 ```plaintext { "DeletedFiles": 3933, "DeletedFolders": 51, "ModifiedFiles": 22, "ExaminedFiles": 2271711, "OpenedFiles": 45685, "AddedFiles": 45663, "SizeOfModifiedFiles": 2606344031, "SizeOfAddedFiles": 2968747713, "SizeOfExaminedFiles": 323371769812, "SizeOfOpenedFiles": 5575100124, "NotProcessedFiles": 0, "AddedFolders": 3395, "TooLargeFiles": 0, "FilesWithError": 0, "ModifiedFolders": 0, "ModifiedSymlinks": 0, "AddedSymlinks": 0, "DeletedSymlinks": 0, "PartialBackup": false, "Dryrun": false, "MainOperation": "Backup", "CompactResults": null, "VacuumResults": null, "DeleteResults": { "DeletedSetsActualLength": 0, "DeletedSets": [], "Dryrun": false, "MainOperation": "Delete", "CompactResults": null, "ParsedResult": "Success", "Interrupted": false, "Version": "2.1.0.2 (2.1.0.2_beta_2024-11-29)", "EndTime": "2025-01-14T18:03:17.2616319Z", "BeginTime": "2025-01-14T18:01:46.1459921Z", "Duration": "00:01:31.1156398", "MessagesActualLength": 0, "WarningsActualLength": 0, "ErrorsActualLength": 0, "Messages": null, "Warnings": null, "Errors": null, "BackendStatistics": { "RemoteCalls": 21, "BytesUploaded": 477998811, "BytesDownloaded": 1002956204, "FilesUploaded": 7, "FilesDownloaded": 12, "FilesDeleted": 0, "FoldersCreated": 0, "RetryAttempts": 0, "UnknownFileSize": 0, "UnknownFileCount": 0, "KnownFileCount": 3613, "KnownFileSize": 189478867337, "LastBackupDate": "2025-01-14T17:32:32+01:00", "BackupListCount": 10, "TotalQuotaSpace": 536870645760, "FreeQuotaSpace": 257338368000, "AssignedQuotaSpace": -1, "ReportedQuotaError": false, "ReportedQuotaWarning": false, "MainOperation": "Backup", "ParsedResult": "Success", "Interrupted": false, "Version": "2.1.0.2 (2.1.0.2_beta_2024-11-29)", "EndTime": "0001-01-01T00:00:00", "BeginTime": "2025-01-14T16:32:32.1808584Z", "Duration": "00:00:00", "MessagesActualLength": 0, "WarningsActualLength": 0, "ErrorsActualLength": 0, "Messages": null, "Warnings": null, "Errors": null } }, "RepairResults": null, "TestResults": { "MainOperation": "Test", "VerificationsActualLength": 12, "Verifications": [ { "Key": "duplicati-20250114T163232Z.dlist.zip.aes", "Value": [] }, { "Key": "duplicati-20241007T141920Z.dlist.zip.aes", "Value": [] }, { "Key": "duplicati-20240425T103610Z.dlist.zip.aes", "Value": [] }, { "Key": "duplicati-20240603T100831Z.dlist.zip.aes", "Value": [] }, { "Key": "duplicati-i3f0213ca535d4291bd4e03ecee3f4bc1.dindex.zip.aes", "Value": [] }, { "Key": "duplicati-i414d196f36fd43be8154bf8b09809135.dindex.zip.aes", "Value": [] }, { "Key": "duplicati-ib973be69caa147e2b63d6022c865bc07.dindex.zip.aes", "Value": [] }, { "Key": "duplicati-i8d0310505ef5425f85a7bcc8d2b5d7cf.dindex.zip.aes", "Value": [] }, { "Key": "duplicati-bb8a709606bc042d9abb081b4f75fce57.dblock.zip.aes", "Value": [] }, { "Key": "duplicati-bbfe54441a1d34fca999398b110d55229.dblock.zip.aes", "Value": [] }, { "Key": "duplicati-b05031f627afa4c129873066dfc87b920.dblock.zip.aes", "Value": [] }, { "Key": "duplicati-bc99e907c69f84df68f12c810f1dadefb.dblock.zip.aes", "Value": [] } ], "ParsedResult": "Success", "Interrupted": false, "Version": "2.1.0.2 (2.1.0.2_beta_2024-11-29)", "EndTime": "2025-01-14T18:03:26.6901016Z", "BeginTime": "2025-01-14T18:03:17.4759097Z", "Duration": "00:00:09.2141919", "MessagesActualLength": 0, "WarningsActualLength": 0, "ErrorsActualLength": 0, "Messages": null, "Warnings": null, "Errors": null, "BackendStatistics": { "RemoteCalls": 21, "BytesUploaded": 477998811, "BytesDownloaded": 1002956204, "FilesUploaded": 7, "FilesDownloaded": 12, "FilesDeleted": 0, "FoldersCreated": 0, "RetryAttempts": 0, "UnknownFileSize": 0, "UnknownFileCount": 0, "KnownFileCount": 3613, "KnownFileSize": 189478867337, "LastBackupDate": "2025-01-14T17:32:32+01:00", "BackupListCount": 10, "TotalQuotaSpace": 536870645760, "FreeQuotaSpace": 257338368000, "AssignedQuotaSpace": -1, "ReportedQuotaError": false, "ReportedQuotaWarning": false, "MainOperation": "Backup", "ParsedResult": "Success", "Interrupted": false, "Version": "2.1.0.2 (2.1.0.2_beta_2024-11-29)", "EndTime": "0001-01-01T00:00:00", "BeginTime": "2025-01-14T16:32:32.1808584Z", "Duration": "00:00:00", "MessagesActualLength": 0, "WarningsActualLength": 0, "ErrorsActualLength": 0, "Messages": null, "Warnings": null, "Errors": null } }, "ParsedResult": "Success", "Interrupted": false, "Version": "2.1.0.2 (2.1.0.2_beta_2024-11-29)", "EndTime": "2025-01-14T18:03:26.7467824Z", "BeginTime": "2025-01-14T16:32:32.1808403Z", "Duration": "01:30:54.5659421", "MessagesActualLength": 51, "WarningsActualLength": 0, "ErrorsActualLength": 0, "Messages": [ "2025-01-14 17:32:32 +01 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started", "2025-01-14 17:39:11 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: ()", "2025-01-14 17:39:11 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: (3.521 KB)", "2025-01-14 17:39:12 +01 - [Information-Duplicati.Library.Main.Operation.BackupHandler-SkipUsnForVolume]: Performing full scan for volume \"C:\\\": USN journal entries were purged since last scan", "2025-01-14 18:52:42 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-bc60c7f2bc8304e95a68024576103397d.dblock.zip.aes (100.240 MB)", "2025-01-14 18:52:43 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-bc60c7f2bc8304e95a68024576103397d.dblock.zip.aes (100.240 MB)", "2025-01-14 18:52:43 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-i8f917ad352cb4789915b6de9e0588ed6.dindex.zip.aes (205.872 KB)", "2025-01-14 18:52:43 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-i8f917ad352cb4789915b6de9e0588ed6.dindex.zip.aes (205.872 KB)", "2025-01-14 18:53:17 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b92c9a69056694435ae2e1a60b965d4ec.dblock.zip.aes (100.147 MB)", "2025-01-14 18:53:27 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b92c9a69056694435ae2e1a60b965d4ec.dblock.zip.aes (100.147 MB)", "2025-01-14 18:53:27 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-i1e218601a2784240878c72c8f64f419c.dindex.zip.aes (153.216 KB)", "2025-01-14 18:53:27 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-i1e218601a2784240878c72c8f64f419c.dindex.zip.aes (153.216 KB)", "2025-01-14 18:54:33 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b76f8fe448e51425f908686efaaf3298a.dblock.zip.aes (86.376 MB)", "2025-01-14 18:55:17 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b76f8fe448e51425f908686efaaf3298a.dblock.zip.aes (86.376 MB)", "2025-01-14 19:00:32 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-i88cfe82eeda94b9daefc4b9b48ddb27f.dindex.zip.aes (332.372 KB)", "2025-01-14 19:00:32 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-i88cfe82eeda94b9daefc4b9b48ddb27f.dindex.zip.aes (332.372 KB)", "2025-01-14 19:01:45 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-20250114T163232Z.dlist.zip.aes (168.417 MB)", "2025-01-14 19:01:46 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-20250114T163232Z.dlist.zip.aes (168.417 MB)", "2025-01-14 19:01:46 +01 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-StartCheck]: Start checking if backups can be removed", "2025-01-14 19:01:46 +01 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-FramesAndIntervals]: Time frames and intervals pairs: 7.00:00:00 / 1.00:00:00, 28.00:00:00 / 7.00:00:00, 365.00:00:00 / 31.00:00:00" ], "Warnings": [], "Errors": [], "BackendStatistics": { "RemoteCalls": 21, "BytesUploaded": 477998811, "BytesDownloaded": 1002956204, "FilesUploaded": 7, "FilesDownloaded": 12, "FilesDeleted": 0, "FoldersCreated": 0, "RetryAttempts": 0, "UnknownFileSize": 0, "UnknownFileCount": 0, "KnownFileCount": 3613, "KnownFileSize": 189478867337, "LastBackupDate": "2025-01-14T17:32:32+01:00", "BackupListCount": 10, "TotalQuotaSpace": 536870645760, "FreeQuotaSpace": 257338368000, "AssignedQuotaSpace": -1, "ReportedQuotaError": false, "ReportedQuotaWarning": false, "MainOperation": "Backup", "ParsedResult": "Success", "Interrupted": false, "Version": "2.1.0.2 (2.1.0.2_beta_2024-11-29)", "EndTime": "0001-01-01T00:00:00", "BeginTime": "2025-01-14T16:32:32.1808584Z", "Duration": "00:00:00", "MessagesActualLength": 0, "WarningsActualLength": 0, "ErrorsActualLength": 0, "Messages": null, "Warnings": null, "Errors": null } } ``` Note that the second example is from the second job run of that day. The first job run took considerably longer but may be non-representative because I moved some folders during the run, leading to about 200 warnings blowing the log up. I do also have verbose logs available if useful but due to the log level each job run results in millions of entries and I would need guidance on analyzing them myself or on how to reliably remove sensitive data from them before sharing. My questions regarding the 2.1.0.2 beta release are: - Are there any changes in the backup procedure known to potentially increase the processing time for my configuration? (I know this question is quite generic) - Am I the only one experiencing the increase in processing time after updating? - Is there a chance that the performance will go back to nomal after migrating to beta 2.1.0.3? If you are confident the issue may disappear when switching to the most recent beta, I don’t want anybody to spend their precious time in investigating legacy issues. If relevant information are missing in my post, feel free to request them. Thanks for your help! ---
**Author:** ![redusr](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/r/bcef8e/32.png) [@redusr](https://forum.duplicati.com/u/redusr)\ **Post date:** [January 27, 2025, 4:24pm UTC](https://forum.duplicati.com/t/decline-in-performance-after-update-to-2-1-0-2-beta/20024/2 "2025-01-27T16:24:43Z")
For the second job, the last pre-update log seems to have already been purged so I can only offer an example after updating to 2.1.0.2: > Begin 2025-01-27 13:59:00 > End 2025-01-27 15:57:26 > Duration 01:58:26 ```plaintext { "DeletedFiles": 108055, "DeletedFolders": 4841, "ModifiedFiles": 2966, "ExaminedFiles": 2274306, "OpenedFiles": 102256, "AddedFiles": 99290, "SizeOfModifiedFiles": 8245212940, "SizeOfAddedFiles": 42863379113, "SizeOfExaminedFiles": 337149099435, "SizeOfOpenedFiles": 51121036065, "NotProcessedFiles": 0, "AddedFolders": 5807, "TooLargeFiles": 0, "FilesWithError": 0, "ModifiedFolders": 0, "ModifiedSymlinks": 0, "AddedSymlinks": 0, "DeletedSymlinks": 0, "PartialBackup": false, "Dryrun": false, "MainOperation": "Backup", "CompactResults": null, "VacuumResults": null, "DeleteResults": { "DeletedSetsActualLength": 0, "DeletedSets": [], "Dryrun": false, "MainOperation": "Delete", "CompactResults": null, "ParsedResult": "Success", "Interrupted": false, "Version": "2.1.0.2 (2.1.0.2_beta_2024-11-29)", "EndTime": "2025-01-27T14:57:16.3087363Z", "BeginTime": "2025-01-27T14:55:56.5475575Z", "Duration": "00:01:19.7611788", "MessagesActualLength": 0, "WarningsActualLength": 0, "ErrorsActualLength": 0, "Messages": null, "Warnings": null, "Errors": null, "BackendStatistics": { "RemoteCalls": 97, "BytesUploaded": 4389042615, "BytesDownloaded": 938648332, "FilesUploaded": 83, "FilesDownloaded": 12, "FilesDeleted": 0, "FoldersCreated": 0, "RetryAttempts": 0, "UnknownFileSize": 0, "UnknownFileCount": 0, "KnownFileCount": 3528, "KnownFileSize": 183686744392, "LastBackupDate": "2025-01-27T13:59:00+01:00", "BackupListCount": 10, "TotalQuotaSpace": 2147483381760, "FreeQuotaSpace": 1752878080000, "AssignedQuotaSpace": -1, "ReportedQuotaError": false, "ReportedQuotaWarning": false, "MainOperation": "Backup", "ParsedResult": "Success", "Interrupted": false, "Version": "2.1.0.2 (2.1.0.2_beta_2024-11-29)", "EndTime": "0001-01-01T00:00:00", "BeginTime": "2025-01-27T12:59:00.0252144Z", "Duration": "00:00:00", "MessagesActualLength": 0, "WarningsActualLength": 0, "ErrorsActualLength": 0, "Messages": null, "Warnings": null, "Errors": null } }, "RepairResults": null, "TestResults": { "MainOperation": "Test", "VerificationsActualLength": 12, "Verifications": [ { "Key": "duplicati-20250127T125900Z.dlist.zip.aes", "Value": [] }, { "Key": "duplicati-20240322T160050Z.dlist.zip.aes", "Value": [] }, { "Key": "duplicati-20241112T110513Z.dlist.zip.aes", "Value": [] }, { "Key": "duplicati-20240716T122326Z.dlist.zip.aes", "Value": [] }, { "Key": "duplicati-i09f1e73e782c4bc3b27f00268344f4fe.dindex.zip.aes", "Value": [] }, { "Key": "duplicati-i7471fb6999964ce2b81b625eb04de14d.dindex.zip.aes", "Value": [] }, { "Key": "duplicati-i406321ef2f8d47949af1c3ef53d3a286.dindex.zip.aes", "Value": [] }, { "Key": "duplicati-i5189c4962f864fddb99134c4555fe605.dindex.zip.aes", "Value": [] }, { "Key": "duplicati-bacf0f015d9a64d32b04d54c8f7dc22ea.dblock.zip.aes", "Value": [] }, { "Key": "duplicati-b899f8dbd3e93414cae1bae0a5bc54536.dblock.zip.aes", "Value": [] }, { "Key": "duplicati-b4cb1d5f340914612be6228ce798727be.dblock.zip.aes", "Value": [] }, { "Key": "duplicati-be1401e848ead4141916906f13b3e78e2.dblock.zip.aes", "Value": [] } ], "ParsedResult": "Success", "Interrupted": false, "Version": "2.1.0.2 (2.1.0.2_beta_2024-11-29)", "EndTime": "2025-01-27T14:57:26.0909635Z", "BeginTime": "2025-01-27T14:57:17.971619Z", "Duration": "00:00:08.1193445", "MessagesActualLength": 0, "WarningsActualLength": 0, "ErrorsActualLength": 0, "Messages": null, "Warnings": null, "Errors": null, "BackendStatistics": { "RemoteCalls": 97, "BytesUploaded": 4389042615, "BytesDownloaded": 938648332, "FilesUploaded": 83, "FilesDownloaded": 12, "FilesDeleted": 0, "FoldersCreated": 0, "RetryAttempts": 0, "UnknownFileSize": 0, "UnknownFileCount": 0, "KnownFileCount": 3528, "KnownFileSize": 183686744392, "LastBackupDate": "2025-01-27T13:59:00+01:00", "BackupListCount": 10, "TotalQuotaSpace": 2147483381760, "FreeQuotaSpace": 1752878080000, "AssignedQuotaSpace": -1, "ReportedQuotaError": false, "ReportedQuotaWarning": false, "MainOperation": "Backup", "ParsedResult": "Success", "Interrupted": false, "Version": "2.1.0.2 (2.1.0.2_beta_2024-11-29)", "EndTime": "0001-01-01T00:00:00", "BeginTime": "2025-01-27T12:59:00.0252144Z", "Duration": "00:00:00", "MessagesActualLength": 0, "WarningsActualLength": 0, "ErrorsActualLength": 0, "Messages": null, "Warnings": null, "Errors": null } }, "ParsedResult": "Warning", "Interrupted": false, "Version": "2.1.0.2 (2.1.0.2_beta_2024-11-29)", "EndTime": "2025-01-27T14:57:26.1898069Z", "BeginTime": "2025-01-27T12:59:00.0244254Z", "Duration": "01:58:26.1653815", "MessagesActualLength": 203, "WarningsActualLength": 1, "ErrorsActualLength": 0, "Messages": [ "2025-01-27 13:59:00 +01 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started", "2025-01-27 14:06:06 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: ()", "2025-01-27 14:06:10 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: (3.364 KB)", "2025-01-27 14:06:11 +01 - [Information-Duplicati.Library.Main.Operation.BackupHandler-SkipUsnForVolume]: Performing full scan for volume \"C:\\\": USN journal entries were purged since last scan", "2025-01-27 14:12:21 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-be9f6c04ae98a40c0ac6179a1e6138aa2.dblock.zip.aes (99.683 MB)", "2025-01-27 14:12:25 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-be11c11b117a34d9d9a441c4c43eacc9d.dblock.zip.aes (99.664 MB)", "2025-01-27 14:12:27 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b7f70b377875546b999e816d9f818f3ab.dblock.zip.aes (99.732 MB)", "2025-01-27 14:12:31 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-bd800cdad759e4e36a73c907f2d980a5f.dblock.zip.aes (99.997 MB)", "2025-01-27 14:12:36 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b7f70b377875546b999e816d9f818f3ab.dblock.zip.aes (99.732 MB)", "2025-01-27 14:12:36 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-be9f6c04ae98a40c0ac6179a1e6138aa2.dblock.zip.aes (99.683 MB)", "2025-01-27 14:12:36 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-be11c11b117a34d9d9a441c4c43eacc9d.dblock.zip.aes (99.664 MB)", "2025-01-27 14:12:36 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-ic2a69fbae4dd43308969aa8e9c0e3c33.dindex.zip.aes (23.122 KB)", "2025-01-27 14:12:36 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-ic2a69fbae4dd43308969aa8e9c0e3c33.dindex.zip.aes (23.122 KB)", "2025-01-27 14:12:36 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b2f1e294f4e904c6896909308efd7cd61.dblock.zip.aes (99.673 MB)", "2025-01-27 14:12:36 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-i2873e7bdf32e43b6b3b7d8973b8db5b9.dindex.zip.aes (24.388 KB)", "2025-01-27 14:12:36 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-bd800cdad759e4e36a73c907f2d980a5f.dblock.zip.aes (99.997 MB)", "2025-01-27 14:12:36 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-ie1dbfee05ee64ebf9d563d52982e715a.dindex.zip.aes (20.372 KB)", "2025-01-27 14:12:36 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-ie1dbfee05ee64ebf9d563d52982e715a.dindex.zip.aes (20.372 KB)", "2025-01-27 14:12:36 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-i2873e7bdf32e43b6b3b7d8973b8db5b9.dindex.zip.aes (24.388 KB)", "2025-01-27 14:12:36 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b53aa94bdd8c44e39a265e6f1f339167f.dblock.zip.aes (99.671 MB)" ], "Warnings": [ "2025-01-27 15:31:15 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\\Users\\user\\Nc\\path\\to\\temporary\\~$2025-01_presentation.pptx\r\nIOException: The process cannot access the file '\\\\?\\C:\\Users\\user\\Nc\\path\\to\\temporary\\~$2025-01_presentation.pptx' because it is being used by another process." ], "Errors": [], "BackendStatistics": { "RemoteCalls": 97, "BytesUploaded": 4389042615, "BytesDownloaded": 938648332, "FilesUploaded": 83, "FilesDownloaded": 12, "FilesDeleted": 0, "FoldersCreated": 0, "RetryAttempts": 0, "UnknownFileSize": 0, "UnknownFileCount": 0, "KnownFileCount": 3528, "KnownFileSize": 183686744392, "LastBackupDate": "2025-01-27T13:59:00+01:00", "BackupListCount": 10, "TotalQuotaSpace": 2147483381760, "FreeQuotaSpace": 1752878080000, "AssignedQuotaSpace": -1, "ReportedQuotaError": false, "ReportedQuotaWarning": false, "MainOperation": "Backup", "ParsedResult": "Success", "Interrupted": false, "Version": "2.1.0.2 (2.1.0.2_beta_2024-11-29)", "EndTime": "0001-01-01T00:00:00", "BeginTime": "2025-01-27T12:59:00.0252144Z", "Duration": "00:00:00", "MessagesActualLength": 0, "WarningsActualLength": 0, "ErrorsActualLength": 0, "Messages": null, "Warnings": null, "Errors": null } } ``` PS: Sorry for double-posting. Including this log in the original post would have exceeded the character limitation for a single post. ---
**Author:** ![kenkendk](https://forum.duplicati.com/user_avatar/forum.duplicati.com/kenkendk/32/5305_2.png) [@kenkendk](https://forum.duplicati.com/u/kenkendk)\ **Post date:** [January 28, 2025, 4:04pm UTC](https://forum.duplicati.com/t/decline-in-performance-after-update-to-2-1-0-2-beta/20024/3 "2025-01-28T16:04:24Z")
> [@redusr](#): > > - Are there any changes in the backup procedure known to potentially increase the processing time for my configuration? (I know this question is quite generic) > - Am I the only one experiencing the increase in processing time after updating? No, quite the opposite. We have spent effort performance optimizing this version, and the reports here on the forum have mentioned speedups. I do not recall any mentions of slowdown. > [@redusr](#): > > Is there a chance that the performance will go back to nomal after migrating to beta 2.1.0.3? Not likely. The 2.1.0.3 is focused on fixing some reported issues. I would be surprised if there was any change in speed. > [@redusr](#): > > If relevant information are missing in my post, feel free to request them. What operating system are you on? And what version of Duplicati have you installed? Is this running in a user context or as a service? Do you have any pragmas or similar configured for the database? ---
**Author:** ![redusr](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/r/bcef8e/32.png) [@redusr](https://forum.duplicati.com/u/redusr)\ **Post date:** [January 28, 2025, 4:40pm UTC](https://forum.duplicati.com/t/decline-in-performance-after-update-to-2-1-0-2-beta/20024/4 "2025-01-28T16:40:02Z")
> [@kenkendk](#): > > No, quite the opposite. We have spent effort performance optimizing this version, and the reports here on the forum have mentioned speedups. I do not recall any mentions of slowdown. Hi @kenkendk , thanks for your reply, that is interesting! Do you have an idea how I could investigate the cause of the decreased performance? > [@kenkendk](#): > > What operating system are you on? > And what version of Duplicati have you installed? > Is this running in a user context or as a service? > Do you have any pragmas or similar configured for the database? - OS: `Win 11 23H2 64-bit` - Duplicati version: `Duplicati - 2.1.0.2_beta_2024-11-29 x64` - Context: I use the `Duplicati.GUI.TrayIcon.exe` and the databases are stored at `%userprofile%\AppData\Local\Duplicati` (should be user-context then, right?) - I do not have any pragmas configured (even though I considered testing that [once](https://forum.duplicati.com/t/how-to-recover-from-aborted-stuck-crashed-backup-run/17818/13) in another context), the database configuration should be as deployed by Duplicati. Not sure whether that is relevant but the only expert options appearing in the job configuration UI are: 1. job 1: `--blocksize=250KB --usn-policy=On --log-file=C:\Users\user\Desktop\job1_log.txt --log-file-log-level=Verbose --no-auto-compact=false` 2. job 2: `--blocksize=400KB --usn-policy=On`. As seen above, blocksize is smaller than recommended (job 2 is newer, that is why it has an increased blocksize) which may have a performance impact but that would not explain the sudden increase after updating. If there is any other info you need, just let me know and I will try to supply. Edit: Added requested info regarding duplicati version. ---
**Author:** ![kenkendk](https://forum.duplicati.com/user_avatar/forum.duplicati.com/kenkendk/32/5305_2.png) [@kenkendk](https://forum.duplicati.com/u/kenkendk)\ **Post date:** [January 28, 2025, 7:55pm UTC](https://forum.duplicati.com/t/decline-in-performance-after-update-to-2-1-0-2-beta/20024/5 "2025-01-28T19:55:16Z")
That all looks like plain settings. There are three architecture versions: x86 (32 bit), x64, and arm64. Are you using the correct package for your system? (I assume you are, but wanted to verify). Another thought was that you could perhaps have some Antivirus/Security product, where you have approved/whitelisted Duplicati 2.0.8.1, and now with the new binary it is not approved and being checked on each file access. This could explain a slowdown, but is it plausible on your system? ---
**Author:** ![redusr](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/r/bcef8e/32.png) [@redusr](https://forum.duplicati.com/u/redusr)\ **Post date:** [January 29, 2025, 10:44am UTC](https://forum.duplicati.com/t/decline-in-performance-after-update-to-2-1-0-2-beta/20024/6 "2025-01-29T10:44:00Z")
> [@kenkendk](#): > > There are three architecture versions: x86 (32 bit), x64, and arm64. > Are you using the correct package for your system? > (I assume you are, but wanted to verify). The system says `64-bit operating system, x64-based processor` and the downloaded installer name is `duplicati-2.1.0.2_beta_2024-11-29-win-x64-gui`, that looks like the right one for the system (I edit-added that info also to my previous post for completeness). > [@kenkendk](#): > > Another thought was that you could perhaps have some Antivirus/Security product, where you have approved/whitelisted Duplicati 2.0.8.1, and now with the new binary it is not approved and being checked on each file access. > > This could explain a slowdown, but is it plausible on your system? The computer uses _Windows Security_ (organization-managed). In the _app- and browser control_ there is a section _exploit protection_. Within that section, _app overrides_ can be defined. There is no override defined for Duplicati. Furthermore, in the _virus and threat protection settings_, _exclusions_ can be defined. I also don’t see Duplicati there as exclusion but this setting is managed by my organization, so I probably never had Duplicati in that exclusion list. Given that the update has overwritten the binary file, I’m also not sure whether I am able to retrace definitely if the previous Duplicati version had an override or exclusion active. Are there specific settings known to interfer with performance that need an override for Duplicati? ---
**Author:** ![kenkendk](https://forum.duplicati.com/user_avatar/forum.duplicati.com/kenkendk/32/5305_2.png) [@kenkendk](https://forum.duplicati.com/u/kenkendk)\ **Post date:** [January 31, 2025, 8:11am UTC](https://forum.duplicati.com/t/decline-in-performance-after-update-to-2-1-0-2-beta/20024/7 "2025-01-31T08:11:07Z")
> [@redusr](#): > > Are there specific settings known to interfer with performance that need an override for Duplicati? No, not directly. But Duplicati is heavy with file access, and many AV products naturally use a detection filter on file access, so this can easily slow down everything… I had another idea while tracking down an unrelated issue. Duplicati 2.0.8.1 uses an integer to decide how many test samples it makes at the end of the backup. This value is called `--backup-test-samples` and has a default value of `1`, meaning that it will test 1 of each (`dlist`, `dindex`, `dblock`). Duplicati 2.1.0.3 has a new setting called `--backup-test-percentage` with a default value of `0.1`, meaning it will sample 0.1% of the remote data after running a backup. If you have a lot of remote files, this could explain a large increase in the processing time, as many additional files are downloaded and tested. If this is the case, you should see the main part of the backup complete as normal, but then spend a long time with the status `Verifying remote data …`. If this is the case, you can revert to the previous way of testing by setting `--backup-test-percentage=0`. ---
**Author:** ![redusr](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/r/bcef8e/32.png) [@redusr](https://forum.duplicati.com/u/redusr)\ **Post date:** [February 5, 2025, 11:04am UTC](https://forum.duplicati.com/t/decline-in-performance-after-update-to-2-1-0-2-beta/20024/8 "2025-02-05T11:04:26Z")
Thanks again for your help! > [@kenkendk](#): > > No, not directly. But Duplicati is heavy with file access, and many AV products naturally use a detection filter on file access, so this can easily slow down everything… Okay, so one try might be to completely whitelist Duplicati and see how the run time changes. > [@kenkendk](#): > > I had another idea while tracking down an unrelated issue. Duplicati 2.0.8.1 uses an integer to decide how many test samples it makes at the end of the backup. This value is called `--backup-test-samples` and has a default value of `1`, meaning that it will test 1 of each (`dlist`, `dindex`, `dblock`). > > Duplicati 2.1.0.3 has a new setting called `--backup-test-percentage` with a default value of `0.1`, meaning it will sample 0.1% of the remote data after running a backup. If you have a lot of remote files, this could explain a large increase in the processing time, as many additional files are downloaded and tested. > > If this is the case, you should see the main part of the backup complete as normal, but then spend a long time with the status `Verifying remote data …`. > > If this is the case, you can revert to the previous way of testing by setting `--backup-test-percentage=0`. I ran job 1 yesterday (no modifications yet) and watched the progress bar status text as well as the live log. According to that, about 10 minutes passed between the initial verification and the check for changes. After another 67 minutes it started putting (e.g., dblock.zip.aes) files and after 6 more minutes `VerifyConsistency` was logged and took a bit more than a minute. There may be potential to improve performance by setting the parameter `--backup-test-percentage=0` but it seems that the largest amount of time by far is spent during checking the source for changes (messages such as the ones below): ```plaintext Including path as no filters matched Skipped checking file, because no metadata was updated ``` Here is the log summary: ```plaintext { "DeletedFiles": 14944, "DeletedFolders": 5, "ModifiedFiles": 3875, "ExaminedFiles": 2272695, "OpenedFiles": 19803, "AddedFiles": 15928, "SizeOfModifiedFiles": 8332688304, "SizeOfAddedFiles": 1715457719, "SizeOfExaminedFiles": 326485205366, "SizeOfOpenedFiles": 10158136125, "NotProcessedFiles": 0, "AddedFolders": 1099, "TooLargeFiles": 0, "FilesWithError": 0, "ModifiedFolders": 0, "ModifiedSymlinks": 0, "AddedSymlinks": 0, "DeletedSymlinks": 0, "PartialBackup": false, "Dryrun": false, "MainOperation": "Backup", "CompactResults": { "DeletedFileCount": 0, "DownloadedFileCount": 0, "UploadedFileCount": 0, "DeletedFileSize": 0, "DownloadedFileSize": 0, "UploadedFileSize": 0, "Dryrun": false, "VacuumResults": null, "MainOperation": "Compact", "ParsedResult": "Success", "Interrupted": false, "Version": "2.1.0.2 (2.1.0.2_beta_2024-11-29)", "EndTime": "2025-02-04T15:10:28.7168682Z", "BeginTime": "2025-02-04T15:06:29.7155228Z", "Duration": "00:03:59.0013454", "MessagesActualLength": 0, "WarningsActualLength": 0, "ErrorsActualLength": 0, "Messages": null, "Warnings": null, "Errors": null, "BackendStatistics": { "RemoteCalls": 71, "BytesUploaded": 2913810571, "BytesDownloaded": 1113444396, "FilesUploaded": 55, "FilesDownloaded": 12, "FilesDeleted": 2, "FoldersCreated": 0, "RetryAttempts": 0, "UnknownFileSize": 0, "UnknownFileCount": 0, "KnownFileCount": 3666, "KnownFileSize": 192173271914, "LastBackupDate": "2025-02-04T14:32:18+01:00", "BackupListCount": 9, "TotalQuotaSpace": 536870645760, "FreeQuotaSpace": 254643855360, "AssignedQuotaSpace": -1, "ReportedQuotaError": false, "ReportedQuotaWarning": false, "MainOperation": "Backup", "ParsedResult": "Success", "Interrupted": false, "Version": "2.1.0.2 (2.1.0.2_beta_2024-11-29)", "EndTime": "0001-01-01T00:00:00", "BeginTime": "2025-02-04T13:32:18.1436606Z", "Duration": "00:00:00", "MessagesActualLength": 0, "WarningsActualLength": 0, "ErrorsActualLength": 0, "Messages": null, "Warnings": null, "Errors": null } }, "VacuumResults": null, "DeleteResults": { "DeletedSetsActualLength": 2, "DeletedSets": [ { "Item1": 1, "Item2": "2025-01-14T17:32:32+01:00" }, { "Item1": 10, "Item2": "2024-01-31T18:35:25+01:00" } ], "Dryrun": false, "MainOperation": "Delete", "CompactResults": { "DeletedFileCount": 0, "DownloadedFileCount": 0, "UploadedFileCount": 0, "DeletedFileSize": 0, "DownloadedFileSize": 0, "UploadedFileSize": 0, "Dryrun": false, "VacuumResults": null, "MainOperation": "Compact", "ParsedResult": "Success", "Interrupted": false, "Version": "2.1.0.2 (2.1.0.2_beta_2024-11-29)", "EndTime": "2025-02-04T15:10:28.7168682Z", "BeginTime": "2025-02-04T15:06:29.7155228Z", "Duration": "00:03:59.0013454", "MessagesActualLength": 0, "WarningsActualLength": 0, "ErrorsActualLength": 0, "Messages": null, "Warnings": null, "Errors": null, "BackendStatistics": { "RemoteCalls": 71, "BytesUploaded": 2913810571, "BytesDownloaded": 1113444396, "FilesUploaded": 55, "FilesDownloaded": 12, "FilesDeleted": 2, "FoldersCreated": 0, "RetryAttempts": 0, "UnknownFileSize": 0, "UnknownFileCount": 0, "KnownFileCount": 3666, "KnownFileSize": 192173271914, "LastBackupDate": "2025-02-04T14:32:18+01:00", "BackupListCount": 9, "TotalQuotaSpace": 536870645760, "FreeQuotaSpace": 254643855360, "AssignedQuotaSpace": -1, "ReportedQuotaError": false, "ReportedQuotaWarning": false, "MainOperation": "Backup", "ParsedResult": "Success", "Interrupted": false, "Version": "2.1.0.2 (2.1.0.2_beta_2024-11-29)", "EndTime": "0001-01-01T00:00:00", "BeginTime": "2025-02-04T13:32:18.1436606Z", "Duration": "00:00:00", "MessagesActualLength": 0, "WarningsActualLength": 0, "ErrorsActualLength": 0, "Messages": null, "Warnings": null, "Errors": null } }, "ParsedResult": "Success", "Interrupted": false, "Version": "2.1.0.2 (2.1.0.2_beta_2024-11-29)", "EndTime": "2025-02-04T15:10:28.7173202Z", "BeginTime": "2025-02-04T15:03:46.5885239Z", "Duration": "00:06:42.1287963", "MessagesActualLength": 0, "WarningsActualLength": 0, "ErrorsActualLength": 0, "Messages": null, "Warnings": null, "Errors": null, "BackendStatistics": { "RemoteCalls": 71, "BytesUploaded": 2913810571, "BytesDownloaded": 1113444396, "FilesUploaded": 55, "FilesDownloaded": 12, "FilesDeleted": 2, "FoldersCreated": 0, "RetryAttempts": 0, "UnknownFileSize": 0, "UnknownFileCount": 0, "KnownFileCount": 3666, "KnownFileSize": 192173271914, "LastBackupDate": "2025-02-04T14:32:18+01:00", "BackupListCount": 9, "TotalQuotaSpace": 536870645760, "FreeQuotaSpace": 254643855360, "AssignedQuotaSpace": -1, "ReportedQuotaError": false, "ReportedQuotaWarning": false, "MainOperation": "Backup", "ParsedResult": "Success", "Interrupted": false, "Version": "2.1.0.2 (2.1.0.2_beta_2024-11-29)", "EndTime": "0001-01-01T00:00:00", "BeginTime": "2025-02-04T13:32:18.1436606Z", "Duration": "00:00:00", "MessagesActualLength": 0, "WarningsActualLength": 0, "ErrorsActualLength": 0, "Messages": null, "Warnings": null, "Errors": null } }, "RepairResults": null, "TestResults": { "MainOperation": "Test", "VerificationsActualLength": 12, "Verifications": [ { "Key": "duplicati-20250204T133218Z.dlist.zip.aes", "Value": [] }, { "Key": "duplicati-20241120T120144Z.dlist.zip.aes", "Value": [] }, { "Key": "duplicati-20240709T131059Z.dlist.zip.aes", "Value": [] }, { "Key": "duplicati-20250114T142202Z.dlist.zip.aes", "Value": [] }, { "Key": "duplicati-i002aa1fce9a4476aad721b7d679e45a7.dindex.zip.aes", "Value": [] }, { "Key": "duplicati-ifca5979c29b44fe190e5965060ec73fd.dindex.zip.aes", "Value": [] }, { "Key": "duplicati-idb7ae29fbb09464482405bc993480b6c.dindex.zip.aes", "Value": [] }, { "Key": "duplicati-i4c9ad38101454e28ad3a0a1cf6e9b16a.dindex.zip.aes", "Value": [] }, { "Key": "duplicati-be6cccd1a66364d179b70d69880f4561c.dblock.zip.aes", "Value": [] }, { "Key": "duplicati-bb5f3d4aab9014d238053505502012233.dblock.zip.aes", "Value": [] }, { "Key": "duplicati-b0c3bff41d1ad4a3683eb896e87243829.dblock.zip.aes", "Value": [] }, { "Key": "duplicati-b26ecd2db4ffd4e38a914c88345d8d0d7.dblock.zip.aes", "Value": [] } ], "ParsedResult": "Success", "Interrupted": false, "Version": "2.1.0.2 (2.1.0.2_beta_2024-11-29)", "EndTime": "2025-02-04T15:10:42.5494373Z", "BeginTime": "2025-02-04T15:10:32.7813976Z", "Duration": "00:00:09.7680397", "MessagesActualLength": 0, "WarningsActualLength": 0, "ErrorsActualLength": 0, "Messages": null, "Warnings": null, "Errors": null, "BackendStatistics": { "RemoteCalls": 71, "BytesUploaded": 2913810571, "BytesDownloaded": 1113444396, "FilesUploaded": 55, "FilesDownloaded": 12, "FilesDeleted": 2, "FoldersCreated": 0, "RetryAttempts": 0, "UnknownFileSize": 0, "UnknownFileCount": 0, "KnownFileCount": 3666, "KnownFileSize": 192173271914, "LastBackupDate": "2025-02-04T14:32:18+01:00", "BackupListCount": 9, "TotalQuotaSpace": 536870645760, "FreeQuotaSpace": 254643855360, "AssignedQuotaSpace": -1, "ReportedQuotaError": false, "ReportedQuotaWarning": false, "MainOperation": "Backup", "ParsedResult": "Success", "Interrupted": false, "Version": "2.1.0.2 (2.1.0.2_beta_2024-11-29)", "EndTime": "0001-01-01T00:00:00", "BeginTime": "2025-02-04T13:32:18.1436606Z", "Duration": "00:00:00", "MessagesActualLength": 0, "WarningsActualLength": 0, "ErrorsActualLength": 0, "Messages": null, "Warnings": null, "Errors": null } }, "ParsedResult": "Success", "Interrupted": false, "Version": "2.1.0.2 (2.1.0.2_beta_2024-11-29)", "EndTime": "2025-02-04T15:10:42.6414866Z", "BeginTime": "2025-02-04T13:32:18.1429166Z", "Duration": "01:38:24.4985700", "MessagesActualLength": 153, "WarningsActualLength": 0, "ErrorsActualLength": 0, "Messages": [ "2025-02-04 14:32:18 +01 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started", "2025-02-04 14:41:36 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: ()", "2025-02-04 14:41:40 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: (3.528 KB)", "2025-02-04 14:41:40 +01 - [Information-Duplicati.Library.Main.Operation.BackupHandler-SkipUsnForVolume]: Performing full scan for volume \"C:\\\": USN journal entries were purged since last scan", "2025-02-04 15:48:58 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-bc62bede6c55d434795caa0e2d570d564.dblock.zip.aes (100.178 MB)", "2025-02-04 15:48:58 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-bc4570e2b4c1c467295fe81b21eddcb41.dblock.zip.aes (100.175 MB)", "2025-02-04 15:48:58 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-bec9411d9233b4e49809c1a7badcc99a6.dblock.zip.aes (100.401 MB)", "2025-02-04 15:48:58 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-bd324ae27eb7a4740aac8bb277cbfaab2.dblock.zip.aes (100.335 MB)", "2025-02-04 15:49:07 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-bc62bede6c55d434795caa0e2d570d564.dblock.zip.aes (100.178 MB)", "2025-02-04 15:49:08 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-bec9411d9233b4e49809c1a7badcc99a6.dblock.zip.aes (100.401 MB)", "2025-02-04 15:49:08 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-bc4570e2b4c1c467295fe81b21eddcb41.dblock.zip.aes (100.175 MB)", "2025-02-04 15:49:08 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-bd324ae27eb7a4740aac8bb277cbfaab2.dblock.zip.aes (100.335 MB)", "2025-02-04 15:49:08 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-ie9bb806ee65844b2a84c248238f78fa5.dindex.zip.aes (196.106 KB)", "2025-02-04 15:49:08 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-i45a06cc33f6a439ab9c13c3186a319ec.dindex.zip.aes (195.528 KB)", "2025-02-04 15:49:08 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-i40f4d07a73ad4a75bcb9201d6b36d715.dindex.zip.aes (536.653 KB)", "2025-02-04 15:49:08 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-i4fae1ae6565a4992bfe6e6772a5b3afe.dindex.zip.aes (202.981 KB)", "2025-02-04 15:49:09 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-ie9bb806ee65844b2a84c248238f78fa5.dindex.zip.aes (196.106 KB)", "2025-02-04 15:49:09 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b96f864ce1b4a49c9ad14348ecba10742.dblock.zip.aes (100.158 MB)", "2025-02-04 15:49:09 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-i45a06cc33f6a439ab9c13c3186a319ec.dindex.zip.aes (195.528 KB)", "2025-02-04 15:49:09 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b466cbf785c22402aa23cd54ecd0cc335.dblock.zip.aes (100.323 MB)" ], "Warnings": [], "Errors": [], "BackendStatistics": { "RemoteCalls": 71, "BytesUploaded": 2913810571, "BytesDownloaded": 1113444396, "FilesUploaded": 55, "FilesDownloaded": 12, "FilesDeleted": 2, "FoldersCreated": 0, "RetryAttempts": 0, "UnknownFileSize": 0, "UnknownFileCount": 0, "KnownFileCount": 3666, "KnownFileSize": 192173271914, "LastBackupDate": "2025-02-04T14:32:18+01:00", "BackupListCount": 9, "TotalQuotaSpace": 536870645760, "FreeQuotaSpace": 254643855360, "AssignedQuotaSpace": -1, "ReportedQuotaError": false, "ReportedQuotaWarning": false, "MainOperation": "Backup", "ParsedResult": "Success", "Interrupted": false, "Version": "2.1.0.2 (2.1.0.2_beta_2024-11-29)", "EndTime": "0001-01-01T00:00:00", "BeginTime": "2025-02-04T13:32:18.1436606Z", "Duration": "00:00:00", "MessagesActualLength": 0, "WarningsActualLength": 0, "ErrorsActualLength": 0, "Messages": null, "Warnings": null, "Errors": null } } ``` Is the log metric `"FilesDownloaded"` indicating the number of verified files (in the UI also visible as formatted text in the _Job → Reporting → Show Log → General → Test Phase → Verifications_ section)? It says ` "FilesDownloaded": 12,` / `Verifications 12` which sounds like a well managable number of files. That is why I would suspect the process of checking for changes in source to account for the decrease in performance. In another [thread](https://forum.duplicati.com/t/how-to-recover-from-aborted-stuck-crashed-backup-run/17818/13), ts678 mentioned the possibility of an SQLite cache bottleneck: > [@How to recover from aborted/stuck/crashed backup run?](https://forum.duplicati.com/t/how-to-recover-from-aborted-stuck-crashed-backup-run/17818/13): > > > [@How to recover from aborted/stuck/crashed backup run?](https://forum.duplicati.com/t/how-to-recover-from-aborted-stuck-crashed-backup-run/17818/12): > > > > for cache size there exists an environment variable I could modify. If I know its name and which value format it expects, I can test increasing the cache size. > > `CUSTOMSQLITEOPTIONS_DUPLICATI` is its name. Was `EXPERIMENTAL_CUSTOMSQLITEOPTIONS_DUPLICATI` and you can probably find several usages under one or the other in the forum and possibly GitHub Issues. > > `set CUSTOMSQLITEOPTIONS_DUPLICATI=cache_size=-200000` is how a batch file gets 200 MB of cache. > Command Prompt and manual launch can also be done. Changing environment variables for a Windows service is awkward compared with a Linux service where it’s very easy. Windows might need a global set. Even though I would not tend to blame SQLite for the sudden decrease in performance, would you recommend testing a larger cache size? Could the full scan message in the log have to do with the long processing times? I think it should not because it was also present in the last log prior to updating, but “full scan” sounds like a lot work to do… ```plaintext "2025-02-04 14:41:40 +01 - [Information-Duplicati.Library.Main.Operation.BackupHandler-SkipUsnForVolume]: Performing full scan for volume \"C:\\\": USN journal entries were purged since last scan", ``` PS: This may be unrelated because I never documented the processing time for that action in the old Duplicati version but maybe it is also relevant for you: The UI control _Operations → Restore files_ also seems to take lots of time to retrieve the data necessary to show the file explorer-alike restore interface. The verbose log shows the start time of each list operation: ```plaintext 2025-02-05 11:43:17 +01 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation List has started 2025-02-05 11:43:17 +01 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation List has started 2025-02-05 11:48:47 +01 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation List has started ``` The waiting time is about 5 minutes until this loading screen appears (time between message 2 and 3): ![image](https://forum.duplicati.com/uploads/default/original/2X/2/29af6eeda0bc5b301d4ba203658066f87383af2d.png) Afterwards, it takes another 4 minutes to build up the first hierarchy level within the root directory `C:\` (this unfortunately is not visible in the log because Duplicati then waits for user interaction before doing anything else). ---
**Author:** ![kenkendk](https://forum.duplicati.com/user_avatar/forum.duplicati.com/kenkendk/32/5305_2.png) [@kenkendk](https://forum.duplicati.com/u/kenkendk)\ **Post date:** [February 5, 2025, 5:37pm UTC](https://forum.duplicati.com/t/decline-in-performance-after-update-to-2-1-0-2-beta/20024/9 "2025-02-05T17:37:19Z")
> [@redusr](#): > > I ran job 1 yesterday (no modifications yet) and watched the progress bar status text as well as the live log. According to that, about 10 minutes passed between the initial verification and the check for changes. After another 67 minutes it started putting (e.g., dblock.zip.aes) files and after 6 more minutes `VerifyConsistency` was logged and took a bit more than a minute. > There may be potential to improve performance by setting the parameter `--backup-test-percentage=0` but it seems that the largest amount of time by far is spent during checking the source for changes (messages such as the ones below): I have also registered a new reported issue related to metadata changes: > > > \> I tried downgrading to v2.1.0.2\_beta but it was still trying to update metadat…a in everyone of my files so I went with the harder option and downgraded back to 2.0.8.1\_beta\_2024-05-07 that I was on originally. Now everything is working great again and I can backup my 1.4 TB in around 5 minutes. > \> > \> \Image > > \_Originally posted by @klieber in \[#5924\](https://github.com/duplicati/duplicati/issues/5924#issuecomment-2629166635)\_ Is this something you see in your setup as well? I have so far not seen it locally on my setups. > [@redusr](#): > > PS: This may be unrelated because I never documented the processing time for that action in the old Duplicati version but maybe it is also relevant for you: The UI control _Operations → Restore files_ also seems to take lots of time Yes, that is unrelated. It is a problem with the database query: > > > When searching a large database it appears to be performing poorly. > > Details a…re in this forum post: > https://forum.duplicati.com/t/taking-too-long-to-list-folders-just-to-find-and-restore-a-handful-of-files/19538 > > The search in the UI is per version, but we should look at handling cross-version searching as well. We will work on this very soon. ---
**Author:** ![redusr](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/r/bcef8e/32.png) [@redusr](https://forum.duplicati.com/u/redusr)\ **Post date:** [February 6, 2025, 12:27pm UTC](https://forum.duplicati.com/t/decline-in-performance-after-update-to-2-1-0-2-beta/20024/10 "2025-02-06T12:27:02Z")
> [@kenkendk](#): > > I have also registered a new reported issue related to metadata changes: > > [v2.1.0.2-v2.1.0.4 incorrectly identifies metadata as changed · Issue #5943 · duplicati/duplicati · GitHub](https://github.com/duplicati/duplicati/issues/5943) > > Is this something you see in your setup as well? I have so far not seen it locally on my setups. Could you describe what the erroneous behavior would look like? Many `metadatachanged` entries that appear unjustified or that say `metadatachanged: True` but the same timestamp in the message two times? I do not see examples in the issue but there seems to be an example in another issue [linked](https://github.com/duplicati/duplicati/issues/5924#issuecomment-2635682062) to it. Using the term `metadatachanged`, I find 19000 hits in the log file for the job run on Tuesday. I can of course not check all of them manually. Some appear plausible (e.g., session-specific files of a few applications which may indeed change over time), for others I am not 100% sure (e.g., `new: True, timestamp changed: True, size changed: True, metadatachanged: True, 2025-01-15 10:37:40 AM vs 0001-01-01 12:00:00 AM` - probably legitimate because file was created after previous backup, the `vs` Timestamp looks quirky but I guess `0001-01-01 12:00:00 AM` is the default timestamp printed for newly created files that cannot have a `vs` Timestamp yet?). At first glance that appears unsuspicious to me. If you have any hints regarding the specific pattern I should be looking for, let me know and I will check and report back. > [@kenkendk](#): > > Yes, that is unrelated. It is a problem with the database query: > > [Improve search speed · Issue #5690 · duplicati/duplicati · GitHub](https://github.com/duplicati/duplicati/issues/5690) > > We will work on this very soon. Good to know, thank you! ---
**Author:** ![kenkendk](https://forum.duplicati.com/user_avatar/forum.duplicati.com/kenkendk/32/5305_2.png) [@kenkendk](https://forum.duplicati.com/u/kenkendk)\ **Post date:** [February 10, 2025, 9:17pm UTC](https://forum.duplicati.com/t/decline-in-performance-after-update-to-2-1-0-2-beta/20024/11 "2025-02-10T21:17:28Z")
> [@redusr](#): > > `new: True, timestamp changed: True, size changed: True, metadatachanged: True, 2025-01-15 10:37:40 AM vs 0001-01-01 12:00:00 AM` Since you have `new: True` that indicates a newly created file. I think that is a legitimate change. The issue I was thinking this could be would be files with: ```plaintext new: False, timestamp changed: False, size changed: False, metadatachanged: True, 2025-01-15 10:37:40 AM vs 2025-01-15 10:37:40 AM ``` So _only_ the metadata has changed here. Can you search the logs for: ```plaintext new: False, timestamp changed: False, size changed: False, metadatachanged: True ``` And see if you have many hits? I guess a few could happen for good reasons, but there should not be many without an explanation. ---
**Author:** ![redusr](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/r/bcef8e/32.png) [@redusr](https://forum.duplicati.com/u/redusr)\ **Post date:** [February 13, 2025, 3:15pm UTC](https://forum.duplicati.com/t/decline-in-performance-after-update-to-2-1-0-2-beta/20024/12 "2025-02-13T15:15:09Z")
> [@kenkendk](#): > > The issue I was thinking this could be would be files with: > > ```plaintext > new: False, timestamp changed: False, size changed: False, metadatachanged: True, 2025-01-15 10:37:40 AM vs 2025-01-15 10:37:40 AM > > ``` > > So _only_ the metadata has changed here. Can you search the logs for: > > ```plaintext > new: False, timestamp changed: False, size changed: False, metadatachanged: True > > ``` > > And see if you have many hits? I guess a few could happen for good reasons, but there should not be many without an explanation. Sure! In the job run on 04th Feb., I found 72 occurrences of that search string. They come from files in the _safebrowsing_ folder of the used firefox profile in _%userprofile%\AppData\Local_ . There are additional entries for older job runs which might or might not be useful for debugging the linked issue: ```plaintext 2025-01-14 16:25:12 +01 - [Verbose-Duplicati.Library.Main.Operation.Backup.FilePreFilterProcess.FileEntry-CheckFileForChanges]: Checking file for changes C:\Users\user\Documents\VSCW\src\annotations\mapping.csv, new: False, timestamp changed: False, size changed: False, metadatachanged: True, 2024-08-23 5:16:09 PM vs 2024-08-23 5:16:09 PM 2025-01-14 16:25:12 +01 - [Verbose-Duplicati.Library.Main.Operation.Backup.FilePreFilterProcess.FileEntry-CheckFileForChanges]: Checking file for changes C:\Users\user\Documents\VSCW\src\annotations\exp.csv, new: False, timestamp changed: False, size changed: False, metadatachanged: True, 2024-08-23 5:16:09 PM vs 2024-08-23 5:16:09 PM ``` The metadatachanged detection is in so far correct, that besides the _annotations_ folder, there is also an _\_annotations_ folder (I swap the names of both folders regularly to have different input configs for a report). The reported files in both folders are content-wise identical but had initially been duplicated from _annotations_ to _\_annotations_ folder. Thus, their creation dates differ between the two folders. After swapping the folder names, Duplicati probably cannot know that another (yet content-wise identical) instance of each file is in the scanned _annotations_ directory and thus assumes a metadata change. What still appears to be strange is that the same message does not occur for the files in the _\_annotations_ folder. Even though I do not know whether the firefox related messages mentioned in the beginning are legitimate, I think they are not the (sole) cause of the performance issue. In the logs prior to the update I found comparable messages (at least for the _safebrowsing_ folder), so at least for my installation I do not see a difference in behavior between the two Duplicati versions in that regard. > [@How to recover from aborted/stuck/crashed backup run?](https://forum.duplicati.com/t/how-to-recover-from-aborted-stuck-crashed-backup-run/17818/13): > > In another [thread](https://forum.duplicati.com/t/how-to-recover-from-aborted-stuck-crashed-backup-run/17818/13), ts678 mentioned the possibility of an SQLite cache bottleneck: > > > [@How to recover from aborted/stuck/crashed backup run?](https://forum.duplicati.com/t/how-to-recover-from-aborted-stuck-crashed-backup-run/17818/13): > > > > > [@How to recover from aborted/stuck/crashed backup run?](https://forum.duplicati.com/t/how-to-recover-from-aborted-stuck-crashed-backup-run/17818/12): > > > > > > for cache size there exists an environment variable I could modify. If I know its name and which value format it expects, I can test increasing the cache size. > > > > `CUSTOMSQLITEOPTIONS_DUPLICATI` is its name. Was `EXPERIMENTAL_CUSTOMSQLITEOPTIONS_DUPLICATI` and you can probably find several usages under one or the other in the forum and possibly GitHub Issues. > > > > `set CUSTOMSQLITEOPTIONS_DUPLICATI=cache_size=-200000` is how a batch file gets 200 MB of cache. > > Command Prompt and manual launch can also be done. Changing environment variables for a Windows service is awkward compared with a Linux service where it’s very easy. Windows might need a global set. > > Even though I would not tend to blame SQLite for the sudden decrease in performance, would you recommend testing a larger cache size? I know that this should not be the cause for a sudden performance drop after updating, but I tested today the effect of increasing the cache size to `20 MB`. I’m glad to report that the processing time was hugely reduced from about 2 hours on Jan. 27 to about 40 minutes today. I will test whether increasing the cache size even more will bring me back to the processing times I was used to prior to the update. > [@kenkendk](#): > > > [@redusr](#): > > > > Are there specific settings known to interfer with performance that need an override for Duplicati? > > No, not directly. But Duplicati is heavy with file access, and many AV products naturally use a detection filter on file access, so this can easily slow down everything… That one could also still be verified - can you tell me which file(s) would be necessary to be excluded from AV rules? Is it the `Duplicati.GUI.TrayIcon.exe`? Even though the setup is more satisfying now, let me know if there are other things I should test. I also consider migrating to the first _stable release 2.1.0.4_ . Can you tell me how the upgrade path looks like for different source releases (_2.0.8.1 beta_ & _2.1.0.2 beta_)? Is there a need for or an advantage of updating incrementally? Should I follow the update instructions in the GUI? Or can I just download the stable release installer and run it, no matter which version (and from which channel) was previously installed? Should I change the release channel to _“stable”_ prior to updating? ---
**Author:** ![kenkendk](https://forum.duplicati.com/user_avatar/forum.duplicati.com/kenkendk/32/5305_2.png) [@kenkendk](https://forum.duplicati.com/u/kenkendk)\ **Post date:** [February 14, 2025, 8:10pm UTC](https://forum.duplicati.com/t/decline-in-performance-after-update-to-2-1-0-2-beta/20024/13 "2025-02-14T20:10:23Z")
> [@redusr](#): > > Even though I do not know whether the firefox related messages mentioned in the beginning are legitimate, I think they are not the (sole) cause of the performance issue. I agree. Only few files, and (assuming) small files. Even scanning these files on each run should not have any substantial impact on the total time used. > [@redusr](#): > > but I tested today the effect of increasing the cache size to `20 MB`. I’m glad to report that the processing time was hugely reduced from about 2 hours on Jan. 27 to about 40 minutes today. Very interesting finding! We were not able to significantly speed up the process through the lookups, but maybe there is something here. @carljohnsen, something to investigate? > [@redusr](#): > > Is it the `Duplicati.GUI.TrayIcon.exe`? Yes, either that or `Duplicati.Server.exe` if you are running as a service. > [@redusr](#): > > Should I change the release channel to _“stable”_ prior to updating? The 2.1.0.3 beta and 2.1.0.4 stable versions are from an identical codebase, except a timeout change for WebDAV, so I would not expect any difference. ---
**Author:** ![ts678](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/t/8491ac/32.png) [@ts678](https://forum.duplicati.com/u/ts678)\ **Post date:** [February 14, 2025, 8:27pm UTC](https://forum.duplicati.com/t/decline-in-performance-after-update-to-2-1-0-2-beta/20024/14 "2025-02-14T20:27:31Z")
> [@kenkendk](#): > > through the lookups Does that refer to the now-retired `use-block-cache` option? That was a very specific use. Personally I think a 2 MB SQLite cache for a database that could be GBs is way too small. What you see happening if you watch system calls (e.g. Process Monitor on Windows) is a tremendous rate of reads through SQLite files, sometimes looking like a linear read of it all. Looking at load on the drive, you’d think no reads were done, but OS cache satisfied them. This doesn’t directly explain why 2.1.0.2 is different, but it was a performance issue before. ---
**Author:** ![kenkendk](https://forum.duplicati.com/user_avatar/forum.duplicati.com/kenkendk/32/5305_2.png) [@kenkendk](https://forum.duplicati.com/u/kenkendk)\ **Post date:** [February 14, 2025, 9:49pm UTC](https://forum.duplicati.com/t/decline-in-performance-after-update-to-2-1-0-2-beta/20024/15 "2025-02-14T21:49:17Z")
> [@ts678](#): > > Does that refer to the now-retired `use-block-cache` option? That was a very specific use. Not directly. @carljohnsen made a number of alternative implementations for some of the lookups, including a fully in-memory dictionary for measuring/experimenting. It was faster, but not by a lot. Several other factors contributed much more to the overall run time, but the results reported by @redusr indicates that we might be able to tweak the database use anyway. ---
**Author:** ![redusr](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/r/bcef8e/32.png) [@redusr](https://forum.duplicati.com/u/redusr)\ **Post date:** [February 18, 2025, 3:41pm UTC](https://forum.duplicati.com/t/decline-in-performance-after-update-to-2-1-0-2-beta/20024/16 "2025-02-18T15:41:55Z")
> [@kenkendk](#): > > > [@redusr](#): > > > > Is it the `Duplicati.GUI.TrayIcon.exe`? > > Yes, either that or `Duplicati.Server.exe` if you are running as a service. > > > [@redusr](#): > > > > Should I change the release channel to _“stable”_ prior to updating? > > The 2.1.0.3 beta and 2.1.0.4 stable versions are from an identical codebase, except a timeout change for WebDAV, so I would not expect any difference. Thanks, will try both in subsequent runs (if I’m patient enough, before separate job runs). I also increased SQLite cache for the other job but this time to 40 MB, processing time decreased from about 100 minutes (Feb. 4) to about 35 minutes (today). I guess 20 MB cache already suffice and the additional cache did no more help but due to testing different jobs with different destination drives (even if config is similar and original job processing times also were), I cannot prove that. The local DB size of the jobs is btw about 3 GB.