Email report to include list of files changed

davegold · September 17, 2017, 4:09am

I’m trying to get my email report to include a list of files.

I’m not seeing this option is the job–I’ve tried verbose, debug, etc, but I don’t think those are what I want them to be.

So I’m creating a post script to run a compare, and then directing the email to use that file as input.

The option send-mail-body says that this option can be a filename, but all I’m getting in the email is the name of the filename, not the next.

So while my specific question is whether anyone has gotten the “send-mail-body” option to include the text of a filename, the real question is whether anyone has gotten a list of files to be included in the email notification.

JonMikelV · September 17, 2017, 4:26am

In what way is the --verbose Advanced option not doing what you want?

Verbose: Use this option to increase the amount of output generated when running an option. Generally this option will produce a line for each file processed.

Is it maybe showing ALL files (as in “processed” means anything even looked at)?

davegold · September 17, 2017, 1:19pm

Hi,

Thats what I thought, but it doesn’t seem to be working.

Here are my job options:

--log-file=d:\log\duplicati.log
--log-retention=7D
--log-level=Information
--send-mail-any-operation=true
--send-mail-to=xxx@domain.com
--send-mail-from=xxx@domain.com
--auto-cleanup=true
--full-result=true
--verbose=true
--send-mail-body=%RESULT%
--run-script-after=filechangelist.bat

And here is the email that I receive:

DeletedFiles: 0
DeletedFolders: 0
ModifiedFiles: 2
ExaminedFiles: 33173
OpenedFiles: 2
AddedFiles: 0
SizeOfModifiedFiles: 35470
SizeOfAddedFiles: 0
SizeOfExaminedFiles: 44323198037
SizeOfOpenedFiles: 35470
NotProcessedFiles: 0
AddedFolders: 0
TooLargeFiles: 0
FilesWithError: 0
ModifiedFolders: 0
ModifiedSymlinks: 0
AddedSymlinks: 0
DeletedSymlinks: 0
PartialBackup: False
Dryrun: False
MainOperation: Backup
ParsedResult: Success
VerboseOutput: True
VerboseErrors: False
EndTime: 9/17/2017 4:19:32 AM
BeginTime: 9/17/2017 4:17:43 AM
Duration: 00:01:48.6972011
Messages: [
    No remote filesets were deleted,
    Compacting not required
]
Warnings: []
Errors: []

I would love if this was user error. Can you see anything I’m doing wrong here?

Dave

dcurrey · September 17, 2017, 4:57pm

I added the following to one of my backup sets.

–full-result=true
–verbose=true

And I am seeing the same basic report as before.

Looks like @davegold output above.

Somebody have a clip of what a verbose output should look like.

kenkendk · September 18, 2017, 9:27am

The options --full-result and --verbose apply to the local copy of the results. If you go to “Log” and then expand the “Result” you should see more data.

I decided that the list of files can be so large that it would be problematic to send it by email (I have seen compressed file lists larger than 4GiB).

If you want the file list to be included in full, some code needs to be added.

This line expands the result object to a string for inclusion in the email:

github.com

duplicati/duplicati/blob/master/Duplicati/Library/Modules/Builtin/SendMail.cs#L458


input = Regex.Replace(input, "\\%PARSEDRESULT\\%", m_parsedresultlevel ?? "", RegexOptions.IgnoreCase | RegexOptions.CultureInvariant);

if (subjectline)

{

    input = Regex.Replace(input, "\\%RESULT\\%", m_parsedresultlevel ?? "", RegexOptions.IgnoreCase | RegexOptions.CultureInvariant);

}

else

{

    if (input.IndexOf("%RESULT%", StringComparison.OrdinalIgnoreCase) >= 0)

        using(TempFile tf = new TempFile())

        {

            RunScript.SerializeResult(tf, result);

            input = Regex.Replace(input, "\\%RESULT\\%", System.IO.File.ReadAllText(tf), RegexOptions.IgnoreCase | RegexOptions.CultureInvariant);

        }

}



foreach (KeyValuePair<string, string> kv in m_options)

    input = Regex.Replace(input, "\\%" + kv.Key + "\\%", kv.Value ?? "", RegexOptions.IgnoreCase | RegexOptions.CultureInvariant);



if (!m_options.ContainsKey("backup-name"))

    input = Regex.Replace(input, "\\%backup-name\\%", System.IO.Path.GetFileNameWithoutExtension(Duplicati.Library.Utility.Utility.getEntryAssembly().Location) ?? "", RegexOptions.IgnoreCase | RegexOptions.CultureInvariant);

We can add something like %FILES% that gets expanded to the list of files. It is slightly difficult to get the full file list because it is not kept in memory.

davegold · September 18, 2017, 11:09am

@kenkendk I appreciate the thorough reply.
The other item to note is that specifying a file as the contents of the email doesn’t seem to be working.

My workaround was to specify a post script that runs a compare, which will give me the info I’m looking for–but when I set the backup job to include that path, it just notes the filename.
Is that something I log in github?

kenkendk · September 18, 2017, 11:12am

Yes, you can create an issue. I looked at the code, and it requires an absolute path to the file:

github.com

duplicati/duplicati/blob/master/Duplicati/Library/Modules/Builtin/SendMail.cs#L291


      
          Exception lastEx = null;

          string lastServer = null;

          

          foreach (var server in servers)

          {

              if (lastEx != null)

                  Logging.Log.WriteWarningMessage(LOGTAG, "SendMailFailedWillRetry", lastEx, Strings.SendMail.SendMailFailedRetryError(lastServer, lastEx.Message, server));

          

              lastServer = server;

              try

              {

                  using (MemoryStream ms = new MemoryStream())

                  {

                      try

                      {

                          using (var client = new SmtpClient(new MailKit.ProtocolLogger(ms)))

                          {

                              client.Timeout = (int)TimeSpan.FromMinutes(1).TotalMilliseconds;

          

                              // Backward compatibility fix for setup prior to using MailKit

                              var uri = new System.Uri(server);

JonMikelV · September 18, 2017, 3:20pm

If you have the time, would you mind sharing the script that you use? It sounds like it could be pretty useful!

I’m hoping it’s prettier than what I’ve tested.

"C:\Program Files\Duplicati 2"\Duplicati.Commandline compare" ssh://mySSH:12345/mnt/Duplicati-Test-Dest?auth-username=me&auth-password=mySshPass&ssh-fingerprint=myFinger" --passphrase=myDuplicatiPass --verbose --send-mail-from=Me@MyDomain.com --send-mail-to=Me@MyDomain.com --send-mail-url=smtp://mySmtpServer:25 --send-mail-any-operation=true

davegold · September 22, 2017, 12:37am

@JonMikelV I set run-script-after of filechangelist.bat, which is in the Duplicati bin directory and includes:

"c:\program files\duplicati 2\Duplicati.CommandLine.exe" compare 0 --dbpath="C:\Users\user1\AppData\Local\Duplicati\YZHGIGIWZG.sqlite" --passphrase=xxxxx --disable-module=console-password-input >d:\log\backuplist.txt

I tried to set send-mail-body to include d:\log\backuplist.txt, but it isn’t pulling text from that file, just listing the file name.

The only thing I can think of is that I’m running Duplicati as a service. I need to test again with a non-service install to verify before I open a ticket in github

update: tested this on another system, and I got it to work, both in as a service account and as a standard install. Seems that the key is to only have the file name (full path) instead of also including %RESULT%
I’m still poking at this, but I think my solution is going to be to not use the built in emailing, but rather just set up my own post script.
Not sure what I would log as an enhancement request at this point.

JonMikelV · September 22, 2017, 1:49am

Great catch on the user account (likely file permission) issue!

davegold · September 22, 2017, 1:58am

@JonMikelV I would have thought, but it was actually trying to add to the default %RESULT% additional info. I’m going to see if I can figure out how to export that info as part of my script, since the stats are useful.