--log-file-log-level vs. --log-level (and --log-file-log-filter), and, how many log files are there, and, what are the different logs, and, what is the meaning of the levels

johnvk · January 21, 2019, 10:29pm

EDIT: added @JonMikelV’s corrections (Thanks JonMikeIV!).
EDIT: added @kenkendk’s info (Thanks kenkendk!).

I would like to know more about “logging” because I want to capture enough log data to be useful for debugging / forum posting, but I don’t want to fill up my disk with spew.

EDIT: --log-file=filename --log-file-log-level=verbose seems to be ideal

First, what’s the difference between advanced options --log-file-log-level and --log-level

ANSWER: --log-level is deprecated; use --log-file-log-level (Oh, there’s this I now see)

To quote the manual (as of 01/21/2019):

log-file-log-level

--log-file-log-level = Warning
Log file information level

and

(EDIT: deprecated – @johnvk)

log-level

--log-level = Warning
Specifies the amount of log information to write into the file specified by --log-file.

(EDIT: deprecated – @johnvk)

Ironically, the option that does not have “file” in its name (ie, --log-level), specifically mentions the --log-file option in its description.

Is there some log file that is not specified with --log-file?

ANSWER: No, not that’s affected by any of these options

This post (Logging levels Values) from a year ago (feb 2018) seems to suggest that the --log-level option doesn’t actually work.

Indeed, what I see most often suggested in this form is --log-file with --log-file-log-level eg, most recently here.

ANSWER: (I repeat) --log-level is deprecated; use --log-file-log-level

(EDIT) Furthermore, once --log-file contents are set by --log-file-log-level they can then be filtered down or augmented by --log-file-log-filter

And what are all the different logs? Here’s what I’ve gathered so far:

User-controlled log:
- File created with --log-file. Affected by --log-file-log-level and --log-file-log-filter
Predefined logs:
- Job log file, under the GUI → job details drop-down → Reporting: → Show log… → has:
  - General - always on; hard-coded contents; each logfile has the word Result on it and appears to be a “result log” I guess
  - Remote - always on; hard-coded contents; which appears to be a rolling log of multiple job runs in order.
- Under GUI → About → Show log → there are
  - Stored - which appears to be stored logs from when ‘Live’ was turned on.
  - Live - which is only active if you set a level
~~Any others?~~

Dare I ask which of the advanced log file options apply to which of these various log outputs?

ANSWER: --log-file-log-level and --log-file-log-filter only affect the user-controlled logfile created by --log-file=

And, finally, what are the different levels? This reply #2 (Logging levels Values - #2 by Pectojin) suggests that the --log-file-log-level values are the same as the GUI.

The GUI for “Live” log lists these options in this ~~(apparently random)~~ order:

Disabled
ExplicitOnly
Profiling
Verbose
Retry
Information
DryRun
Warning
Error

Those are not in alphabetical order, and it doesnt look like they’re in verbosity order (I don’t see any way the sequence Disabled … Warning … Error can be consistently increasing or decreasing in verbosity).

Except for the first 2, these are in order of decreasing verbosity. Ie Profiling is the most verbose, and Error is the least verbose.

Some notes about the levels and their order of verbosity:

If you use --log-file-log-level=warning you get warning AND errors.

profiling is the highest and most verbose including all levels below it.

So, profiling outputs more than verbose .

As might be expected, verbose outputs more than information .

ExplicitOnly is special and not normally useful.

Interestingly, DryRun includes Errors and Warnings too. Makes sense I guess.

And Retry includes everything DryRun includes, plus Informationals , plus everything at the Retry level itself.

But both verbose and profiling contain both DryRun and Retry so there might not be much use for those more specific levels.

Using a hint from this reply #4 (OneDrive (personal) back-ups suddenly started failing with "Channel is retired" and "A task has been cancelled" - #4 by ts678), and reply #2 here I list these levels in increasing-verbosity order to the best of my knowledge:

[previous list removed]

~~Those last two look like special cases somehow.~~

~~Or is each of these a flag that get all 'OR’ed together (Ie, print log line if Explicit OR Error OR Retry)?~~

Thanks for all your help so far, and in advance for this one

(Moderators: please feel free to edit this post to correct it and keep it up to date)

JonMikelV · January 21, 2019, 11:40pm

The 4 database logs you listed are they only ones and they always log - there are no parameters (I know if) to adjust their log levels.

The --log-Ievel parameter is deprecated and only works with older versions of Duplicati.

The verbose of the options available for --log-file-log-level varies a bit depending on the action being taken and warnings/errors that happen, but on the whole if say your verbosity order is good except I’d move ExplicitOnly after Warning.

Just to confuse you more, don’t forget the log-filter options that will include/exclude message the regardless of the log-level. I use this for logging JUST database calls, for example.

johnvk · January 22, 2019, 3:31am

Can you proofread my edits?

kenkendk · January 22, 2019, 8:31am

They are intended to be in verbosity order, as defined here:

github.com

duplicati/duplicati/blob/master/Duplicati/Library/Logging/Log.cs#L29-L63


      
          public enum LogMessageType

          {

              /// <summary>

              /// The message should only be shown if it is explicitly requested

              /// </summary>

              ExplicitOnly,

              /// <summary>

              /// The message is a profiling message

              /// </summary>

              Profiling,

              /// <summary>

              /// Messages that are normally not wanted for display

              /// </summary>

              Verbose,

              /// <summary>

              /// The message is from a retry

              /// </summary>

              Retry,

              /// <summary>

              /// The message is informative but does not indicate problems

This file has been truncated. show original

The ExplicitOnly level is special. The messages emitted with the level ExplicitOnly are only emitted if a filter matches them directly, or the level is set to ExplicitOnly. These messages are generally just noise, as they are expected messages (such as failing to load some native libraries during backend loading, etc).

I have done some work trying to make the logging more sane, but a core issue is that the backup itself (which can also run on the commandline) is emitting some log messages, and the server (which is showing the UI, running the tasks, and having the scheduler) is also emitting some messages. I am mentioning this as it might help to understand it until we figure out a better way of presenting the messages.

johnvk · January 28, 2019, 10:46pm

Thanks so much @kenkendk that clears it up.

I will update my OP.

For those who don’t read code, I will state in “English” . In increasing levels of verbosity, the levels are:

Error
Warning
DryRun
Information
Retry
Verbose
Profiling

Meaning, if you use --log-file-log-level=warning you get warning AND errors.

profiling is the highest and most verbose including all levels below it.

So, profiling is more than verbose.

As might be expected, verbose outputs more than information.

ExplicitOnly is special and not normally useful.

Interestingly, DryRun includes Errors and Warnings too. Makes sense I guess.

And Retry includes everything DryRun includes, plus Informationals, plus everything at the Retry level itself.

But both verbose and profiling contain both DryRun and Retry so there might not be much use for those more specific levels.