How to recover from aborted/stuck/crashed backup run?

[ts678]

@redusr

For each of those backup runs, there is only one instance in which both messages do not appear in the same second (but about 5 seconds delayed).

It’s good to hear that the huge unexplainable gap isn’t common. I’m not enough of an expert dev to guess at where your hours of delay came from, although if it comes back again, you have new things you can check.

@redusr

Regarding the PRAGMA, I hope I will not need to use it but it could be a feature request to add an option in the Duplicati Web GUI to run it (maybe alternatively to a full repair?) or to do it even automatically if Duplicati can detect it has quit inproperly.

The forum is not a good issue tracker, but you can certainly see if it’s in GitHub Issues. If not, you can open.
The way PRAGMA optimize is explained, it sounds like it’s very fast if it doesn’t actually have to run analyze. Running one at database opens might be quick, although there’s still a question of how well optimize works.

@redusr

Does that mean that also in case of an inproperly quit Duplicati, log entries up to that point should exist? Or are they affected by the rollback when restarting Duplicati?

The change makes some things that used to go only into the hard-to-find server log also go to usual job log, however I think the job log can still be lost in transaction rollback. Exact commit points are not documented.

@redusr

In the latter case, submitting log entries over a second connection and immediately committing them could maybe help?

SQLite only allows one writer at a time, I think, although I’m not sure when it’s enforced. Maybe at connect?

Also, wouldn’t commit apply to the whole database, including other changes which aren’t ready to commit?

This is something that worries me currently. Lots of concurrency, and each part commits as it feels a need. What happens if a good time for part A is a bad time for part B? Still looking at how the design handles that.

@redusr

in the GUI progress bar / status the currently performed step

The status bar is already too crowded for what it tries to show, so it has to use a variable display sequence. Posting specific internals details of SQL and such seems way too much to show, or to be comprehensible.

@redusr

could be a new/extended entry in About → System info that indicates more accurately then currently where the process is stuck.

This is where I thought the details could be hidden. Your idea of a combination approach might be workable, however it would be up to some developer (and they are scarce) to figure out what might be implemented…

Duplicati 2: Display current task in status

shows some early thinking, although the bar of “a moment” or “a second” seems way too sensitive for large backups. The idea of “what it’s currently doing” is also old, as 2.0.3.6_canary_2018-04-23 went concurrent.

Channel Pipeline

I’ve been trying to wrap my head around the pipeline that performs a backup. Partially to better understand how it decides what files to process and to look at ways to improve the abstract file IO setup since different parts of the app enumerate files differently. I’ve also noticed a significant number of exceptions related to trying to handle a directory as a file which just gets swept under the rug. The other thing that bugs me is this duplicate file enumeration in the pipeline to count files…

I’m still wishing there was a design document on concurrency, with transaction design that fits in properly.

Lacking that, I tried expanding my thinking in the channel pipeline topic into some pipelines. Main might be:

FileEnumerationProcess.cs → SourcePaths → MetadataPreProcess.cs → ProcessedFiles → FilePreFilterProcess.cs → AcceptedChangedFile → FileBlockProcessor.cs → StreamBlock → StreamBlockSplitter.cs → OutputBlocks → DataBlockProcessor.cs → BackendRequest

and several of those (but not yet charted) are doing database actions and occasional commits-as-seen-fit.