Duplicati database rebuild - 2gb client database for 750gb of backup files

Restoring files if your Duplicati installation is lost is likely a faster path because it only builds what it calls a “partial temporary database”, presumably tailored to the request, but not worrying about all versions of all files. Database recreate is a known slow spot. There are some timing measurements going on right now…

Database recreate performance

although I’m not sure where it will head. A rewrite of some of this area has also begun, but I’m not sure if it may bring increased effectiveness in actually getting issues fixed, increased performance, or maybe both.

I wonder if Profiling log info builds up? If so, possibly using –log-file and –log-file-log-level would avoid this.

Aside from download time (which is somewhat unavoidable, but there might also be issues hanging around that cause dblock files to be downloaded when actually only smaller dlist and dindex files are required), the lengthy operations I found were all SQL INSERT operations. The Block and BlocksetEntry tables track quite small (by default) 100KB deduplication blocks. Choosing sizes in Duplicati covers the settings and tradeoff. I’ve got no specific values to suggest, based on testing, but at least one user seems to have experimented.

There are other ugly answers. Some people back up the database in a secondary backup (seems perhaps overkill for 22GB source, but people with TBs of source might want it). Some decide to start a fresh backup. That’s what I did once, but I don’t really care about old versions. Other people really want to preserve them.

I think situation is not what it should be (so for now things above may be needed) but it’s not being ignored.