# DB Query Performance Testing, Fixes, and Maintainability **URL:** \ **Category:** Developer\ **Created:** [February 27, 2022, 2:46am UTC](https://forum.duplicati.com/t/db-query-performance-testing-fixes-and-maintainability/14018 "2022-02-27T02:46:16Z")\ **Posts on this page:** 1\ **Showing post:** 5
**Author:** ![ts678](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/t/8491ac/32.png) [@ts678](https://forum.duplicati.com/u/ts678)\ **Post date:** [March 15, 2022, 3:27pm UTC](https://forum.duplicati.com/t/db-query-performance-testing-fixes-and-maintainability/14018/5 "2022-03-15T15:27:52Z")
> [@tarianjed](#): > > The whole database recreate is 1 transaction actually. Not sure how I feel about that. I’m not enough of a DB expert to comment, however one thing I worry about is my generally limited understanding of the underlying design philosophy of keeping relationships consistent despite what catastrophes (e.g. hard kills) may occur. Consistency (or at least repairability) need is also external, especially for ongoing backups. Maybe it’s less so for things like DB recreate that can’t be resumed. Processing IIRC does `dlist` files first, so dangling block references will exist until `dindex` readings. Perhaps this order suits [Direct restore from backup files](https://duplicati.readthedocs.io/en/latest/03-using-the-graphical-user-interface/#restoring-files-if-your-duplicati-installation-is-lost) best (where one picks the version initially)? Sorry I’m not familiar with the code internals. Most of my experience has been external observation. [Write-Ahead Logging](https://sqlite.org/wal.html) and [Enable write-ahead logging and memory mapped IO #4612](https://github.com/duplicati/duplicati/pull/4612) may fit in too, especially when talking about performance, and potentially temporary file sizes during DB recreate. In terms of external files that don’t have a transaction model, there’s a [state-tracking](https://github.com/duplicati/duplicati/issues/1243#issuecomment-71367471) mechanism to hopefully make it possible to [clean up](https://github.com/duplicati/duplicati/blob/2f3b0e83a0c3e5dbdd8ccba0a08e3623d64ca231/Duplicati/Library/Main/Operation/FilelistProcessor.cs#L34-L73) whatever sometimes-partially-done work is at the destination. Some of this cleanup, though, relies on the database which in theory has some aids to transactions. Question is, did we use transactions correctly, especially for the concurrent processing that occurs? Sometimes one thread must flush something it wants out, but does that break the plans elsewhere? > [@tarianjed](#): > > I think maybe a hybrid in-memory work database that we persist results from to the disk database is probably the most effective method. Let me know what you think. One challenge that Duplicati faces on memory is the extremely diverse size of systems that it runs on. Maybe a challenging case would be a large backup running on a NAS, perhaps with Duplicati Docker (because Synology as of DSM 7 no longer will install the current version – Docker is the workaround). An excellent person to bring into this would be the hard-to-find original author who also [tried a rewrite](https://forum.duplicati.com/t/repair-is-downloading-dblock-files/2780/11). Unfortunately, it [never emerged](https://forum.duplicati.com/t/excessive-failures-warnings-aka-making-duplicati-more-stable/6828/4). I wonder if you would be a suitable person to try to pick up the effort? It seems a huge waste to discard something possibly so near at least some initial level of “completed”. Original author still seems short on time though, but as someone who’s done lots, you deserve an ask. Your willingness to dig in is fantastic. Maybe you’ll be invited to join staff (which is quite thin right now). @kenkendk what do you think about this seemingly very relevant topic, and also recreate questions? --- _[View the full topic](https://forum.duplicati.com/t/db-query-performance-testing-fixes-and-maintainability/14018)._