Really frustrated with Duplicati's lack of feedback

[kenkendk]

@cyberfox

Anyway, what I was going to ask is this…a common mistake people make when using logging tools (or heaven forfend, writing their own) is that when you’re logging for debugging purposes like profiling, you have to flush after every write. Some logging frameworks allow for ‘unbuffered’ writing, which is just as good. Do you know if the log methods are unbuffered, or flush after each write?

This is “roll-your-own” logging but it does flush after messages, so you can use tail -f <path> to see the progress. Using --log-level=profiling will include a starting message for all queries to catch this particular issue.

You should see a message like this:

Starting - ExecuteQuery: SELECT ....

This will be written and flushed before the actual query is executed.

@cyberfox

The use of sqlite is also problematic, because it doubles down on the lack of visibility into the work actually being done.

I think it was a good choice some 10+ years ago because it keeps the memory usage low and performs relatively fast B+ tree lookups. But it does not scale well with +1TiB backups which are more common today. I am aware of the scaling issues, and intend to work on it soon-ish.