Just installed Duplicati today and running my first backups. This a stand-alone machine backing up to a 5TB USB3 Seagate drive. Windows 10 Pro (22H2) HP-Z440 64GB RAM The first backup set went well and completed in a reasonable amount of time. 20.82GB in 00:15:04 My second set is going much slower, it is significantly larger 1.32TB, 73,172 files, in 1,969 folders. What is troubling is that the throughput is degrading consistently through the process. It was transferring 50MB/s at first and with 57,000 files remaining it is down to 19.45 MB/s and has been running for 7 hours. With only 22% of the files completed and down to less than half of the throughput, I’m wondering if this will complete. I’ll leave it running overnight and hopefully it will. No encryption Volume Size: 100MB Thanks in advance! ::next day… 18 hours into the backup. Throughput seems to have settled at 17 MB/s. 15K files remain. Will a different Volume Size affect this? Is there a way to commit more RAM to the process? And would that help. I see some conversation about Block Size, should that be adjusted? File sizes are mostly around 25MB each. I wanted to run these to my NAS as a parallel backup. But that looks like it could take days to get started and the ability to restore from there seems dubious.

Hi @mbehrens, welcome to the forum :waving_hand: [image] mbehrens: Will a different Volume Size affect this? Most likely not. This is just the size of the generated zip files. [image] mbehrens: Throughput seems to have settled at 17 MB/s. 15K files remain. We (well, mostly @carljohnsen) have looked into performance in this area and it looks like there is a point where inserts into the SQLite database becomes slower and slower. It makes sense that you only see this for the larger backup because the problem is not there until a certain size. What you can do is try with a larger --blocksize which is the internal unit of data that Duplicati processes. If you set this to something like --blocksize=10mb it will make the number of hash entries in the database 10x smaller than the current default (provided most files are +10mb in size).

[image] kenkendk: the problem is not there until a certain size I keep pointing out (based on Process Monitor activity) that Duplicati seems to thrash the SQLite page cache, which defaults to a mere 2 MB of cache for maybe a 4 GB database. CUSTOMSQLITEOPTIONS_DUPLICATI=cache_size=- (note minus sign there) can increase this and make the paging activity less. I hope someone’s testiing this angle, preferably not on something that hides it by being too fast. I had my database on my own 5TB Seagate USB drive for awhile, thinking two spindles (C: is the other) can load-share. Actually the portable drive kind of busies out, and it’s not from rare destination file writing. I’m not sure what cache I had then, but it might have been default, as a Windows service can’t have environment variables changed in it fast in Windows. Duplicati has now added: Preload settings which can nicely make desired changes to Environment variables - env, however adding cache can backfire on a memory limited system. Mine’s 32 GB – and full. My last test had 100 MB of cache using this preload.json file put in the database folder. { "env": { "*": { }, "tray": { }, "server": { "CUSTOMSQLITEOPTIONS_DUPLICATI": "cache_size=-100000" } }, "db": { "server": { } }, "args": { } } I’m not sure which table most affects the cache. Anything blocksize oriented is suspect. For non-initial backups, scan speed is probably also slowed if the file information gets big, requiring holding a lot of names in cache. Maybe the prefix table plan helps? I don’t know.

Thanks for the tips. Changing the blocksize will require me to recreate the backup set. Since the initial backup is completed and the incrementals are running in reasonable amount of time, I’ll keep this in mind for now. If I have to do a restore I understand that this backup set may take a while to complete.

[image] ts678: I keep pointing out (based on Process Monitor activity) that Duplicati seems to thrash the SQLite page cache, which defaults to a mere 2 MB of cache for maybe a 4 GB database. Good point. This can also improve things and is a fairly simple change to apply. Maybe we should even expose this as a setting?

I see you put in the feature request already, along with a start at an auto-tuning scheme. Add option to set SQLite page cache and set defaults based on system memory opened 08:13AM - 18 Apr 25 UTC (UTC) [image] kenkendk minor change performance issue good first issue It is currently possible to set the SQLite page cache with the environment varia…ble: ``` CUSTOMSQLITEOPTIONS_DUPLICATI=cache_size=- ``` However, this requires some lookups into the docs for SQLite before it can be used. Adding an option like `--sqlite-page-cache=10mb` would make it a lot simpler to activate. Proactively, we can check how much memory the system has and choose a sensible default (like 100MiB if the OS has more than 16GiB of ram). This setting should be applied before the custom options, so the user can override any default, as that will make it continue working with existing settings (if the system already has custom options prior to the upgrade). Things that need manual tuning add user work and support load. Either way needs plan. If someone prefers to test on Linux, strace with -P path may sub for Process Monitor. I was just looking for suspicious 4 KB (the default page size) I/Os to database or journal. SQLite C Interface has some SQLITE_DBSTATUS_CACHE_* metrics (and more) to study if C# can find a way in. Putting it in product might be hard, but possibly, someone could write a C tool for dev test. SQLite allows memory mapped I/O, but it has some drawbacks compared to read + write. Architecture of SQLite pictorially summarizes the area I’m worrying about, what it calls the Backend consisting of B-Tree, Pager, and OS interface. Someone can check, but the theory to me is that B-Tree needs pages, and if they’re not in the cache, it goes to OS which has its own cache, but if that fails, then that actually hits a drive. Giving more page cache might also hit a drive, as it’s probably virtual memory, so has the potential to busy (example) the Windows page file. What to assume for drive speed is a good question. Trend for small systems is SSDs, but that’s still costly in large sizes, so it’s a question of what systems Duplicati should support. I’ll note that Duplicati is pretty slow for me compared to other backups on this old desktop.

[image] ts678: Things that need manual tuning add user work and support load. Either way needs plan. The plan for now is to use the system memory as a guide, given that SQLite would take a piece of it. For now I would like to try with a default 1% of total memory as the default. I have a PR for it here: Added option to set the SQLite page cache master ← feature/add-sqlite-cache-value opened 12:28PM - 22 Apr 25 UTC (UTC) [image] kenkendk +304 -83 This adds an extra option to allow setting the SQLite page cache size as a regul…ar option. Prior to this commit it was only possible to set the SQLite page cache size via environment variables. The option to use environment variables is preserved, and as options from the environment variable are applied after the new setting, environment variables take precedence. The default value for the new option is to use 1% of the system memory for the page cache. For the restore process, a connection per worker may be made which defaults to half the number of cores, so the maximum amount of memory is 1% * half the CPU cores. This is just the upper limit, and SQLite may choose not to use all of it. This fixes #6178 Essentially, it will be transparent for the user, and just allow SQLite to use up to 1% of system memory (per connection, backup uses 1, restores can use more). To opt out is simply to set the value to zero. If it is the same or less than the default 2000kb, it just never tries the pragma that sets the cache size (same as before the PR). [image] ts678: Someone can check, but the theory to me is that B-Tree needs pages, and if they’re not in the cache, it goes to OS which has its own cache, but if that fails, then that actually hits a drive. Yes, that is a good description. Each node in the tree exists in pages, so any lookup requires at least one page. The larger the table, the more node lookups are required, but it grows very slowly in height (high fan-out). SQLite can use some heuristics to decide which pages to keep in memory (higher in the tree should be kept longer), but once it needs a particular page that is not in memory, the OS needs to provide it from “storage”. The size option just changes the level at which the OS needs to be involved, so I see this as a change with a low chance of negative impact and a decent chance for positive impact, regardless of the system. [image] ts678: I’ll note that Duplicati is pretty slow for me compared to other backups on this old desktop. Is that with the default cache size, or did you try to increase it?

[image] kenkendk: Is that with the default cache size, or did you try to increase it? Either way. I think my first increase was trying 500 MB (about 10% of the database size). Duplicati’s commit got to about 1400 MB, but it might have become high at default cache. Interestingly, active working set was far lower, but that’s how many programs seem to go. Regardless, I tried 100 MB next, and my tuning-by-process-monitor didn’t look that awful.

Duplicati

Backup Slows Down In Process

Support

show post in topic

Home
Categories
Guidelines
Terms of Service
Privacy Policy

Powered by Discourse, best viewed with JavaScript enabled