Logging: seeking comprehensive understanding

1

I have a long list of questions about logging and how it works that I don’t think are documented fully anywhere…

  1. What kind of information is logged in the Job->Reporting->Show Log area vs the About->Show Log area? Is it job-specific logs vs server-level logs? I’d imagine, but then I see what appears to be job-specific stuff in the latter. (mention of specific blocks being started/completed in the Live section)
  2. For the logs in Job->Reporting->Show Log:
    a. What is the difference between General and Remote? Is it that Remote only pertains to operations related data sent/received to the Job Destination? And General is everything else?
    b. For Remote, is the data coming from the job-specific SQLite file in the RemoteOperation table?
    c. For Remote, is the data ever truncated (old log events deleted)?
    d. For General, it is organized by job run. So are job runs even deleted from this log? When?
    e. For General, it appears to be coming from the job-specific SQLIte file in the LogData table. It uses the JSON in the Message column. Correct?
    f. For General, the Warnings section of a job’s log appears to be truncated, since the number of Warnings listed is far larger than the number shown when this section is expanded. How can I see the entire untruncated list of Warnings? It seems like the JSON in SQLite may also be truncated in this section. Correct? The “Complete Log” appears to show the raw JSON and has Warnings truncated.
  3. For the logs in About->Show Log:
    a. What is the difference between Stored and Live? Totally unintuitive given the difference in what I see in them. Are they supposed to be the same and it’s just that Stored has to be reloaded to see updates and Live is live? Not the case for me where Stored is always older events than Live even when no jobs are running.
    b. Where is the Stored section actually stored? Is it in the server-level SQLite file in the ErrorLog table? Seems to be.
    c. Is the Stored log ever truncated? When?
    d. Is the Live log supposed to be dynamically updating live if I just sit there and watch the page over time? I assume I need to select a Level other than “Disabled” to see that. (Edit: Yes it is dynamic, but readers take note that there seems to be a maximum length to the Live log as shown on the GUI, such that once events are too old or too numerous, they vanish from the Live view as new ones come in)
    e. The Live log level section dropdown seems to have a bug where once you select one level, selecting any other level does nothing. True?
    f. Is the Live log persisted anywhere? Or is it just kept in memory transiently within the server.
  4. I see that a job-specific text log file can also be activated via the --log-file option. This appears to contain Warnings/Errors that are the same as those in General, but not truncated. However the other summary statistics in General are absent. So perhaps this is the answer to my question 2f above.
  5. For all of the logs mentioned above (excluding General), can they be written to files instead of only available in the GUI and SQLite?
  6. For the Stored and Remote logs, can their logging level be changed?
  7. Can the General/Remote logs be viewed live for a currently running job in the GUI? (Edit: It seems like Remote can, but General cannot)
2

Also, I see a popup that says “Failed to connect: database is locked database is locked” when trying to view these logs in the GUI while a job is running sometimes. And the log will say “Loading”. Known bug?

I think this error is reported in the Stored log as follows:

Apr 30, 2022 11:24 PM: Request for http://localhost:8200/api/v1/backup/2/log?pagesize=15 gave error
code = Busy (5), message = System.Data.SQLite.SQLiteException (0x800007AF): database is locked
database is locked
   at System.Data.SQLite.SQLite3.Prepare(SQLiteConnection cnn, String strSql, SQLiteStatement previous, UInt32 timeoutMS, String& strRemain)
   at System.Data.SQLite.SQLiteCommand.BuildNextCommand()
   at System.Data.SQLite.SQLiteDataReader.NextResult()
   at System.Data.SQLite.SQLiteDataReader..ctor(SQLiteCommand cmd, CommandBehavior behave)
   at System.Data.SQLite.SQLiteCommand.ExecuteReader(CommandBehavior behavior)
   at Duplicati.Server.WebServer.RESTMethods.LogData.DumpTable(IDbCommand cmd, String tablename, String pagingfield, String offset_str, String pagesize_str)
   at Duplicati.Server.WebServer.RESTMethods.Backup.FetchLogData(IBackup backup, RequestInfo info)
   at Duplicati.Server.WebServer.RESTHandler.DoProcess(RequestInfo info, String method, String module, String key)
Apr 30, 2022 11:24 PM: Reporting error gave error
System.ObjectDisposedException: Cannot write to a closed TextWriter.
   at System.IO.__Error.WriterClosed()
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at Duplicati.Server.WebServer.RESTHandler.DoProcess(RequestInfo info, String method, String module, String key)
3

That’s definitely a long list. I don’t think there’s any active developer (I’m not one) who could answer all.

Are you interested in documenting? The manual is a separately-owned project, but it might accept pulls.
Because it’s somewhat user (not developer) oriented, translating technical answers (which I might give) into something more user-friendly would be good. Logging is definitely an area that can confuse people.

Would you like some partial answers? What is the need behind the long list?