ah yes, of course.
That’s my opinion as well.
No, thanks again for the effort.
If I try to summarize your results:
dbrecreate saved 15%, 18%, 21%, and (big) for 0, 1, 2 lost dindex and fatal lost dblock respectively.
Normal case savings of 15% is not nothing, but is not huge either. I don’t know if tweaking available indexes could improve it. Maybe someday we’ll get better at analyzing where resources are going…
Speaking of that, I’m testing on just one system, and results might vary, e.g. on a fast one with SSD. Watching more closely with Task Manager and Resource Monitor, the hard drive here got very busy.
Activity level got high in the Temp folder etilqs_<random> files, which I think are SQLite temporaries. Sysinternals Process Monitor was the tool used for that. I/O is usually 4096 bytes, SQLite page size. Currently the screen is showing lots of random reads in one file, but sometimes it’s reads and writes.
I’d also note that I’m testing near the high end of the recommended block count, and so don’t know if lower or dreadfully-higher block counts (shudder) would affect the performance improvement I found.
Good performance testing is a lot of work, and (as always) a willing volunteer could certainly help out.
I did not check the Internet, but my feeling is that the temporary tables are stored there indeed. The interest is obvious: the disk space is automatically given back to the system without having to compact the database.
all my past experience with database makes me think that when queries deteriorate with more data , they are doing so in an exponential way. And well, the accounts by posters on this forum seem to concur.
I have 1.5 M blocs in my biggest test bed now. Tomorrow I will see what difference it makes with a healthy database.
current canary: 42’
dbrecreate pr + expanded cache: 20’
That 52% reduction is similar to the 57% reduction from my test above, lines 2 and 3.
I’m no longer testing line 1 (dblist) recreate, but substituted dbrecr with normal cache.
This allows comparisions betwen the dbrecr and the cache improvements separately.
Combined, of course, is the fastest. Below I switched to testing my typical backup, so
dlist count is higher (about 60), blocksize is 100 KB, and block count is about 256000.
Missing 0 dindex
Processing indexlist volume 214 of 214
Recreate completed, verifying the database consistency
seconds dbrecr dblist cache
409 0 0 0
286 0 0 1
355 1 0 0
277 1 0 1
That’s about a 32% reduction with both enhancements.
Less than the high-block improvement (as you predict).
I installed and tested basic functionality (Backup new, Restore new, Restore existing) on a few Macs and had no issues other than Mac OS complaining about a signed developer. Glad to see this release getting traction!
Thanks for the feedback ! much welcome since I don’t have a Mac to test.
Well, time to wrap up this thing.
I have updated my 3 experimental PR.
The list of PR that will go in is now more or less fixed:
4802 Tray Mac OS
I’ll add a fix to remove a warning on the previous PR I think.
4815 more exclusions
4849 faster hashing fix
4897 Mega library update
4901 SSL library update
4902 Debian bookworm install fix
4903 add back Bouncy castle license
4904 change default deb format to zip to accommodate old Debian distros
4905 Impossible Cloud
4907 Custom Sqlite pragmas
to be created : PR to update Newtonsoft to 13.02 since the Dependabot PR are not enough and conflict with the previous PR.
I’m now going to review the last details before asking the project owner to validate the new release, in the mean time if anyone has an objection it’s still time (to remove a bad PR that is, not to add a new one it’s too late).
4901 SSL library update should say SSH not SSL (PR title could also get this fix)
4907 Custom Sqlite pragmas So only this one gets mention as non-experimental?
4895 EXPERIMENTAL_DBLIST_DUPLICATI
4890 EXPERIMENTAL_RECREATEDB_DUPLICATI
Do experiments go in, minus most publicity? SQL review would be nice, but who will?
I guess you and project owner can debate release notice format, also in changelog.txt
The PR author credits were nice. Also document environment variable as appropriate.
Maybe we keep experimental ones low-key. Official one might someday be an option?
Is there a security-only fix known to remain here now that Newtonsoft has gone away?
Newtonsoft conflicted with custom SQLite pragmas? Or was the reference to another?
I did notice that some Minio or AWSSDK versions moved. Was that dependency pain?
EDIT:
Actually Renci.SshNet.dll going to 2020.0.2 definitely qualifies as a security-only fix.
I updated the typo in SSH PR, thanks for the heads-up.
4907 will go in as a minimal very low risk enhancement, EXPERIMENTAL will stay uncommitted (I did NOT mention them in the list) - even if dbbuild seem to have some potential, it would need more reviews as it’s a significant change in core.
Yes Changelog is to do.
Newtonsoft has not gone away, it’s just that the Dependabot PR are not so useful - for some reason some dependencies were not updated so the end result was that it was not effective -and I intend to replace them with a global one once the other are committed.
I did not consider other libraries to keep things as simple as possible.
It wasn’t clear why below got in. I had guessed maybe a dependency fix?
AWSSDK.Core 3.3.103.37 3.3.103.71
AWSSDK.IdentityManagement 3.3.103.26 3.3.103.52
AWSSDK.S3 3.3.104.25 3.3.104.43
Yes I remember now. See here:
and the following. In the end I gave up and left it at that as I have no way to do actual tests on a Mac.
Thanks. As guessed, it was needed to get the project to build. Next unexpected one I’m noticing is in
EditUriBuiltins.js where something around the storj-satellite option is being changed. I can’t find a PR.
EDIT:
How to search pull requests on Github targeting specific file? was too much, so I wonder if you know?
It was an test that I did not follow up. That’s why I did not do a PR. Basically it was a fix to the UI to allow for another option of the Storj backend to be used because I hoped it would work better than the ‘normal’ one. Instead it was worse. So even if it’s a attempt at a bug fix it’s not one I want to be into Duplicati proper - it’s better for it to stay broken.
Thanks. No more questions beyond some general maybe-future musing about automated test coverage and how well it tests the capabilities enabled by options or environment variables (might be easier to do).
I’m wishing some more people jumped in to try specific fixes, but I guess that will have to wait for Canary.
Wonderful job with this effort, and it will be great to have some of those lost functionality chunks returned.
Thank you, and I hope things go well on the remaining steps leading to release (more process invention).
At first glance, it seems that the only step is writing a changelog and sending it to the project owner.
Actually it’s a bit more complicated than that.
Reading more closely the release script, it is building Duplicati with Gtk enabled, something I have found to be a blocker when using the new Mac Trayicon code. So I will have to change the release script to build Duplicati 2 times, one time for Mac without Gtk enabled, and one time with Gtk enabled for the other platforms, if not, the released Canary will not work correctly for Macs.
That’s not especially difficult - I have done just that with the Github actions - except that I am lacking a Mac and the prerequisites for running the release script in the first place (that is, the auth keys). In short, I’m lacking the particular Mac of the project owner.
So I will have to do some dummying and proxying and hope for a bit of luck in changing this release script. Also I have not a lot of available time for Duplicati this week-end. That’s why it’s not done yet.
That’s all now. As far as I can tell, all that is remaining now is to send a changelog to the project owner and ask to run the (updated) release script.
@ts678
Here is the intended changelog. Do you think I missed something of concern ?
* Disable automatic use of v2 authid for Jottacloud, thanks @albertony
* Fix gui tests, thanks @gpatel-fr
* Make Xamarin-based CocoaRunner the default on Mac, thanks @dgileadi
* Fix #4716 by falling back to System hasher, thanks @vilaureu
* Allow install on Debian bookworm by using libayatana-appindicator1, thanks @gpatel-fr
* Build Debs with gzip compression for old debian releases, thanks @gpatel-fr
* Add Impossible Cloud provider for S3 backend, thanks @daniel0m0baker
* Add possibility to set custom SQlite pragmas, thanks @gpatel-fr
* updates Newtonsoft, Mega and SSH.NET libraries, thanks @gpatel-fr
* raise the file time shift to 2s for Mac unittests, thanks @gpatel-fr
* remove obsolete Letsencrypt cert in Docker builds, thanks @Bubblesaway (forum)
Edited for better formattingLooks good to me except for some stylistic things (and I’m not going to nitpick at ALL capitalizations)
SSH-NET should be SSH.NET.
I suppose sqlite should be SQLite, but more importantly, I guess pragmas are explained on-demand?
That might be the safer path…
I fretted a bit over the Dockerfile PR and double ampersand potential sh short-circuit, but fine for now.
Thanks so much for pushing a step closer to a Canary. Does this mean Mac can no longer .zip-install?
I thought it used to be that all the installers started from the .zip file, but now Mac needs another build?
Edited my previous post as per your remarks, thanks.
Not sure what you mean by ‘explained on-demand’. Changelog is not the place to documentation IMO, do you really want to explain how it’s managed here ?
I tested the Dockerfile change by sligthly editing the build-images.sh (removed the multiplatform capability, removed the ‘rm’ of the build at the end, and changed the output from ‘image, push’ to docker. So I tested the generated image locally, both with and without the fix and verified the change in the output of the csharp command in the resulting container.
About Mac, the change in the build-release.sh file (PR #4916) has been done to address this problem by generating the final output including all relevant toolkits in the TrayIcon executable. I tested that it was effective by running it with the --help option on the Mac Github build machine, and verified that on a Linux computer the tray icon works (in this case --help returns Winforms, Gtk and Cocoa) and also on a Win10 computer (in this case --help returns Winforms and Cocoa). So the TrayIcon should work on a Mac (unless somehow the library is not available on the project owner’s computer - but the build script will stop in this case so this should be detected before shipping an invalid release - if it happens we shall see - fingers crossed and all that).
It all means that a full build can only be done on a Mac, too. But it was already the case anyway.
It’s not ideal. A full explanation would be rather lengthy. We also have little specific advice for it yet.
On-demand meant in forum or issues, as needed, by a person leading some change or experiment.
If preference is the manual, there’s a maintainer shortage. It’s many years behind on some things…
You could probably get write permission to other things like manual, for similar plan to product code.
Generally I worry more about trying to get docs prepared for Beta when code hits bigger distribution.
I forget if a Beta is now ASAP. Is it hoped Canary testers try pragma out? If so, how are they guided?
IIRC we were heading for the long-awaited Beta, but making some other PRs catch Beta after, right?