Date Problems on Backup Report

handyguy · November 8, 2017, 9:32pm

Excellent! Thank you. I will code this into the next version of dR. Appreciate your listening.

silversword411 · November 17, 2017, 5:05am

A little slow, but. Well, you were discussing issues with problems with discussing dates, and after 2 re-readings I now understand what you’re saying because of: “Remember, dR has only the job status email text to go by for all its work. dR has no access to the Duplicati server or database from which to glean any additional intelligence about the backup job or the locale.”

You’re parsing:
EndTime: 11/17/2017 4:46:54 AM
BeginTime: 11/17/2017 4:46:41 AM
…which I think Duplicati will allow the user to customize/and is affected by local settings based on device/os/settings on which Duplicati was run on. I feel sorry for you trying to deal with garbage data input to make a reliable data output.

I was suggesting adding a 2nd sets of dates to the Duplicati log E-mails:
BeginTimeUTC:
EndTimeUTC:
…but you don’t have access to do that as dR

So I guess my solution is still sound, but you’ll have to get those changes in Duplicati and added to the E-mail report, then you’d be able to utilize that by having an always absolute time value to parse in dR!

Niels_Hoogenhout · November 17, 2017, 6:10am

Agree with that, for multiple format support it would be nice if there would be a second output of the date and time without local formatting. Would be great if it could be added to the report.

I’m using the send-http-url advanced option with my own script and was running into the same issue with some family members getting wrong dates shown on their dashboards (and alert emails).

Maybe a bit related, if not feel free to split the next part! Why is the report only available in plain text? Especially for http post, but probably also for local usage, a json string would be a lot easier to handle. Now I’m breaking the text by characters what works kind of oke but could have some issues with special characters. A gzip json string would be smaller to post and faster to process.

handyguy · November 17, 2017, 1:49pm

@kenkendk and I went a few rounds on this issue and I think we landed in a pretty good place.

Currently, dupReport 2.0.4 can handle a wide variety of date formats that come in on the EndTime & BeginTime lines (MM/DD/YYY, DD.MM.YYYY, YYYY-MM-DD, etc.). It cheats a little by having the user specify the expected date format in the .rc file on startup. This value is global for all emails, but in the upcoming 2.1 it will be able to be set per backup set, in case you’re receiving backup emails from multiple locales, each with it’s own date formatting. For now, I’m skirting around the timezone/UTC offset problem by ignoring it: whatever time is presented in the email result is reported back out in the dR report.

For 2.1 I’m considering taking the UTC offset info from the incoming email header:

Date: Tue, 24 Oct 2017 15:47:13 -0400

The “-0400” gives an indication of the UTC offset for the incoming email. I can then apply that offset to the EndTime/BeginTime lines and come up with a more accurate representation of the actual date/time of the backup. That’s the theory, anyway. I’d appreciate any feedback you may have on the soundness of this approach.

Finally, @kenkendk indicated that he was adding a timestamp value into the reports:

EndTime: 11/7/2017 12:25 PM (1483273500)

This would allow dR to more easily parse the line and reverse the timestamp into a local date/time value. I endorsed the idea and am able to work with it. If the Council of Smart People deem it better to have separate ‘TimeUTC’ lines in the email, I can work with that also; it just complicates the parsing of the email a bit and adds a little more if/then coding in dR. I’ll leave it up to the CoSP to make a final determination.

As I’m working on dR 2.1 I’m trying to build in more flexibility to handle the insanity of arcane and inconsistent date & time rules and formats. It hasn’t been easy.

Niels_Hoogenhout · November 17, 2017, 2:09pm

That would indeed be a great addition! Looking forward to his next release with that included.

JonMikelV · November 17, 2017, 7:22pm

If a “standard” for the JSON could be decided on, there’s no reason we couldn’t add an --include-reporting-JSON-in-email parameter.

Or maybe just expose it as a %REPORTING_JSON% variable that could be added to the email body.

Niels_Hoogenhout · November 18, 2017, 7:33pm

Well for email I think using plain text is just fine. That’s what an email eventually is. But for http post it’s a different story since it’s data that’s usually meant to be processed and not just read by an end user.

JSON already has sort of a standardized output, by example:
{"key1":"value1","key2":"value2","keymessages":["message1","messsage2","etc"]}

I’m not familiar with C# unfortunately, but guessing the data is already an array the is probably a json_encode function that could be used.

To be honest I even think the current http-post could be replaced with a well formated way by default. Don’t know if @kenkendk agrees he would probably have to change it. but would make more sense.

JonMikelV · November 19, 2017, 1:49am

Agreed, I mis-stated what I meant. I was trying to say all parties concerned could agree on the JSON contents up front rather than having an implementation followed by updates.

kenkendk · December 9, 2017, 8:51pm

It is a data structure, but it is fairly simple to dump it as JSON.

Niels_Hoogenhout · December 10, 2017, 11:56pm

Thanks for the reply. If it’s already structured data, could you maybe give me a hint how to use it in an easy way, or how to get a json dump.

kenkendk · December 11, 2017, 7:57am

If you are looking at the email, the body contents are generated here:

github.com

duplicati/duplicati/blob/master/Duplicati/Library/Modules/Builtin/SendMail.cs#L458


      
          using System;

          using System.Collections;

          using System.Collections.Generic;

          using System.IO;

          using System.Linq;

          using System.Net;

          using System.Text;

          using System.Text.RegularExpressions;

          using DnsLib;

          using Duplicati.Library.Interface;

          using Duplicati.Library.Logging;

          using Duplicati.Library.Utility;

          using System.Net.NetworkInformation;

          using System.Net.Sockets;

          using MailKit.Net.Smtp;

          using MimeKit;

          using Duplicati.Library.Modules.Builtin.ResultSerialization;

          

          namespace Duplicati.Library.Modules.Builtin

          {

              public class SendMail : ReportHelper

              {

                  /// <summary>

                  /// The tag used for logging

                  /// </summary>

                  private static readonly string LOGTAG = Logging.Log.LogTagFromType<SendMail>();

          

                  #region Option names

          

                  /// <summary>

                  /// Option used to specify server url

                  /// </summary>

                  private const string OPTION_SERVER = "send-mail-url";

                  /// <summary>

                  /// Option used to specify server username

                  /// </summary>

                  private const string OPTION_USERNAME = "send-mail-username";

                  /// <summary>

                  /// Option used to specify server password

                  /// </summary>

                  private const string OPTION_PASSWORD = "send-mail-password";

                  /// <summary>

                  /// Option used to specify sender

                  /// </summary>

                  private const string OPTION_SENDER = "send-mail-from";

                  /// <summary>

                  /// Option used to specify recipient(s)

                  /// </summary>

                  private const string OPTION_RECIPIENT = "send-mail-to";

                  /// <summary>

                  /// Option used to specify mail subject

                  /// </summary>

                  private const string OPTION_SUBJECT = "send-mail-subject";

                  /// <summary>

                  /// Option used to specify mail body

                  /// </summary>

                  private const string OPTION_BODY = "send-mail-body";

                  /// <summary>

                  /// Option used to specify mail level

                  /// </summary>

                  private const string OPTION_SENDLEVEL = "send-mail-level";

                  /// <summary>

                  /// Option used to specify if reports are sent for other operations than backups

                  /// </summary>

                  private const string OPTION_SENDALL = "send-mail-any-operation";

                  /// <summary>

                  /// Option used to specify what format the result is sent in.

                  /// </summary>

                  private const string OPTION_RESULT_FORMAT = "send-mail-result-output-format";

                  /// <summary>

                  /// Option used to set the log level for mail reports

                  /// </summary>

                  private const string OPTION_LOG_LEVEL = "send-mail-log-level";

                  /// <summary>

                  /// Option used to set the log filters for mail reports

                  /// </summary>

                  private const string OPTION_LOG_FILTER = "send-mail-log-filter";

                  /// <summary>

                  /// Option used to set the maximum number of log lines

                  /// </summary>

                  private const string OPTION_MAX_LOG_LINES = "send-mail-max-log-lines";

                  #endregion

          

                  #region Option defaults

                  /// <summary>

                  /// The default mail sender

                  /// </summary>

                  private const string DEFAULT_SENDER = "no-reply";

                  #endregion

          

                  #region Private variables

                  /// <summary>

                  /// The server url to use

                  /// </summary>

                  private string m_server;

                  /// <summary>

                  /// The server username

                  /// </summary>

                  private string m_username;

                  /// <summary>

                  /// The server password

                  /// </summary>

                  private string m_password;

                  /// <summary>

                  /// The mail sender

                  /// </summary>

                  private string m_from;

                  /// <summary>

                  /// The mail recipient

                  /// </summary>

                  private string m_to;

                  #endregion

          

          

                  #region Implementation of IGenericModule

                  /// <summary>

                  /// The module key, used to activate or deactivate the module on the commandline

                  /// </summary>

                  public override string Key { get { return "sendmail"; } }

          

                  /// <summary>

                  /// A localized string describing the module with a friendly name

                  /// </summary>

                  public override string DisplayName { get { return Strings.SendMail.Displayname;} }

          

                  /// <summary>

                  /// A localized description of the module

                  /// </summary>

                  public override string Description { get { return Strings.SendMail.Description; } }

          

                  /// <summary>

                  /// A boolean value that indicates if the module should always be loaded.

                  /// If true, the  user can choose to not load the module by entering the appropriate commandline option.

                  /// If false, the user can choose to load the module by entering the appropriate commandline option.

                  /// </summary>

                  public override bool LoadAsDefault { get { return true; } }

          

                  /// <summary>

                  /// Gets a list of supported commandline arguments

                  /// </summary>

                  public override IList<ICommandLineArgument> SupportedCommands

                  {

                      get

                      {

                          return new List<ICommandLineArgument>(new ICommandLineArgument[] {

                              new CommandLineArgument(OPTION_RECIPIENT, CommandLineArgument.ArgumentType.String, Strings.SendMail.OptionRecipientShort, Strings.SendMail.OptionRecipientLong),

                              new CommandLineArgument(OPTION_SENDER, CommandLineArgument.ArgumentType.String, Strings.SendMail.OptionSenderShort, Strings.SendMail.OptionSenderLong, DEFAULT_SENDER),

                              new CommandLineArgument(OPTION_SUBJECT, CommandLineArgument.ArgumentType.String, Strings.SendMail.OptionSubjectShort, Strings.SendMail.OptionSubjectLong(OPTION_BODY), DEFAULT_SUBJECT),

                              new CommandLineArgument(OPTION_BODY, CommandLineArgument.ArgumentType.String, Strings.SendMail.OptionBodyShort, Strings.SendMail.OptionBodyLong, DEFAULT_BODY),

                              new CommandLineArgument(OPTION_SERVER, CommandLineArgument.ArgumentType.String, Strings.SendMail.OptionServerShort, Strings.SendMail.OptionServerLong),

                              new CommandLineArgument(OPTION_USERNAME, CommandLineArgument.ArgumentType.String, Strings.SendMail.OptionUsernameShort, Strings.SendMail.OptionUsernameLong),

                              new CommandLineArgument(OPTION_PASSWORD, CommandLineArgument.ArgumentType.String, Strings.SendMail.OptionPasswordShort, Strings.SendMail.OptionPasswordLong),

                              new CommandLineArgument(OPTION_SENDLEVEL, CommandLineArgument.ArgumentType.String, Strings.SendMail.OptionSendlevelShort, Strings.SendMail.OptionSendlevelLong(ParsedResultType.Success.ToString(), ParsedResultType.Warning.ToString(), ParsedResultType.Error.ToString(), ParsedResultType.Fatal.ToString(), "All"), DEFAULT_LEVEL, null, Enum.GetNames(typeof(ParsedResultType)).Union(new string [] { "All" }).ToArray()),

                              new CommandLineArgument(OPTION_SENDALL, CommandLineArgument.ArgumentType.Boolean, Strings.SendHttpMessage.SendhttpanyoperationShort, Strings.SendHttpMessage.SendhttpanyoperationLong),

          

                              new CommandLineArgument(OPTION_LOG_LEVEL, CommandLineArgument.ArgumentType.Enumeration, Strings.ReportHelper.OptionLoglevellShort, Strings.ReportHelper.OptionLoglevelLong, DEFAULT_LOG_LEVEL.ToString(), null, Enum.GetNames(typeof(Logging.LogMessageType))),

                              new CommandLineArgument(OPTION_LOG_FILTER, CommandLineArgument.ArgumentType.String, Strings.ReportHelper.OptionLogfilterShort, Strings.ReportHelper.OptionLogfilterLong),

                              new CommandLineArgument(OPTION_MAX_LOG_LINES, CommandLineArgument.ArgumentType.Integer, Strings.ReportHelper.OptionmaxloglinesShort, Strings.ReportHelper.OptionmaxloglinesLong, DEFAULT_LOGLINES.ToString()),

          

                              new CommandLineArgument(OPTION_RESULT_FORMAT, CommandLineArgument.ArgumentType.Enumeration, Strings.ReportHelper.ResultFormatShort, Strings.ReportHelper.ResultFormatLong(Enum.GetNames(typeof(ResultExportFormat))), DEFAULT_EXPORT_FORMAT.ToString(), null, Enum.GetNames(typeof(ResultExportFormat))),

                          });

                      }

                  }

          

                  protected override string SubjectOptionName => OPTION_SUBJECT;

                  protected override string BodyOptionName => OPTION_BODY;

                  protected override string ActionLevelOptionName => OPTION_SENDLEVEL;

                  protected override string ActionOnAnyOperationOptionName => OPTION_SENDALL;

                  protected override string LogLevelOptionName => OPTION_LOG_LEVEL;

                  protected override string LogFilterOptionName => OPTION_LOG_FILTER;

                  protected override string LogLinesOptionName => OPTION_MAX_LOG_LINES;

                  protected override string ResultFormatOptionName => OPTION_RESULT_FORMAT;

          

          		/// <summary>

          		/// This method is the interception where the module can interact with the execution environment and modify the settings.

          		/// </summary>

          		/// <param name="commandlineOptions">A set of commandline options passed to Duplicati</param>

          		protected override bool ConfigureModule(IDictionary<string, string> commandlineOptions)

                  {

                      //We need at least a recipient

                      commandlineOptions.TryGetValue(OPTION_RECIPIENT, out m_to);

                      if (string.IsNullOrEmpty(m_to))

                          return false;

          

                      commandlineOptions.TryGetValue(OPTION_SERVER, out m_server);

                      commandlineOptions.TryGetValue(OPTION_USERNAME, out m_username);

                      commandlineOptions.TryGetValue(OPTION_PASSWORD, out m_password);

                      commandlineOptions.TryGetValue(OPTION_SENDER, out m_from);

          

                      if (string.IsNullOrEmpty(m_from))

                          m_from = DEFAULT_SENDER;

          

                      return true;

                  }

          

                  #endregion

          

                  #region Implementation of IGenericCallbackModule

          

                  /// <summary>

                  /// Sends the email message

                  /// </summary>

                  /// <param name="subject">The subject line.</param>

                  /// <param name="body">The message body.</param>

                  protected override void SendMessage(string subject, string body)

                  {

                      var message = new MimeMessage();

                      MailboxAddress mailbox;

                      foreach (string s in m_to.Split(new[] { "," }, StringSplitOptions.RemoveEmptyEntries))

                          if (MailboxAddress.TryParse(s.Replace("\"", ""), out mailbox))

                              message.To.Add(mailbox);

          

                      var mailboxToFirst = (MailboxAddress)message.To.First();

                      string toMailDomain = mailboxToFirst.Address.Substring(mailboxToFirst.Address.LastIndexOf("@", StringComparison.Ordinal) + 1);

          

                      string from = m_from.Trim().Replace("\"", "");

                      if (from.IndexOf('@') < 0)

                      {

                          if (from.EndsWith(">", StringComparison.Ordinal))

                              from = from.Insert(from.Length - 1, "@" + toMailDomain);

                          else

                              from = string.Format("No Reply - Backup report <{0}@{1}>", from, toMailDomain);

                      }

          

                      if (MailboxAddress.TryParse(from, out mailbox))

                          message.From.Add(mailbox);

          

                      message.Subject = subject;

                      message.Body = new TextPart("plain") { Text = body, ContentTransferEncoding = ContentEncoding.EightBit };

          

                      List<string> servers = null;

                      if (string.IsNullOrEmpty(m_server))

                      {

                          var dnslite = new DnsLib.DnsLite();

                          var dnslist = new List<string>();

          

                          //Grab all IPv4 addresses

                          foreach (NetworkInterface networkInterface in NetworkInterface.GetAllNetworkInterfaces())

                              try

                              {

                                  foreach (IPAddress dnsAddress in networkInterface.GetIPProperties().DnsAddresses)

                                      if (dnsAddress.AddressFamily == AddressFamily.InterNetwork)

                                          dnslist.Add(dnsAddress.ToString());

                              }

                              catch (Exception ex) { Logging.Log.WriteExplicitMessage(LOGTAG, "DNSServerLookupFailure", ex, "Failed to get DNS servers from network interface"); }

          

                          dnslist = dnslist.Distinct().ToList();

          

                          // If we have no DNS servers, try Google and OpenDNS

                          if (dnslist.Count == 0)

                          {

                              // https://developers.google.com/speed/public-dns/

                              dnslist.Add("8.8.8.8");

                              dnslist.Add("8.8.4.4");

          

                              //http://www.opendns.com/opendns-ip-addresses/

                              dnslist.Add("208.67.222.222");

                              dnslist.Add("208.67.220.220");

                          }

          

                          var records = new List<MXRecord>();

                          foreach (var s in dnslist)

                          {

                              var res = dnslite.getMXRecords(toMailDomain, s);

                              if (res != null)

                                  records.AddRange(res.OfType<MXRecord>());

                          }

          

                          servers = records.OrderBy(record => record.preference).Select(x => "smtp://" + x.exchange).Distinct().ToList();

                          if (servers.Count == 0)

                              throw new IOException(Strings.SendMail.FailedToLookupMXServer(OPTION_SERVER));

                      }

                      else

                      {

                          servers = (from n in m_server.Split(new string[] { ";" }, StringSplitOptions.RemoveEmptyEntries)

                                     let srv = (n == null || n.IndexOf("://", StringComparison.OrdinalIgnoreCase) > 0) ? n : "smtp://" + n

                                     where !string.IsNullOrEmpty(srv)

                                     select srv).Distinct().ToList();

                      }

          

                      Exception lastEx = null;

                      string lastServer = null;

          

                      foreach (var server in servers)

                      {

                          if (lastEx != null)

                              Logging.Log.WriteWarningMessage(LOGTAG, "SendMailFailedWillRetry", lastEx, Strings.SendMail.SendMailFailedRetryError(lastServer, lastEx.Message, server));

          

                          lastServer = server;

                          try

                          {

                              using (MemoryStream ms = new MemoryStream())

                              {

                                  try

                                  {

                                      using (var client = new SmtpClient(new MailKit.ProtocolLogger(ms)))

                                      {

                                          client.Timeout = (int)TimeSpan.FromMinutes(1).TotalMilliseconds;

          

                                          // Backward compatibility fix for setup prior to using MailKit

                                          var uri = new System.Uri(server);

                                          if (uri.Scheme.ToLowerInvariant() == "tls")

                                              uri = new System.Uri("smtp://" + uri.Host + ":" + (uri.Port <= 0 ? 587 : uri.Port) + "/?starttls=always");

          

                                          client.Connect(uri);

          

                                          if (!string.IsNullOrEmpty(m_username) && !string.IsNullOrEmpty(m_password))

                                              client.Authenticate(m_username, m_password);

          

                                          client.Send(message);

                                          client.Disconnect(true);

                                      }

                                  }

                                  finally

                                  {

                                      var log = Encoding.UTF8.GetString(ms.GetBuffer());

                                      if (!string.IsNullOrWhiteSpace(log))

                                          Logging.Log.WriteProfilingMessage(LOGTAG, "SendMailResult", Strings.SendMail.SendMailLog(log));

                                  }

                              }

          

                              lastEx = null;

                              Logging.Log.WriteInformationMessage(LOGTAG, "SendMailComplete", Strings.SendMail.SendMailSuccess(server));

                              break;

                          }

                          catch (Exception ex)

                          {

                              lastEx = ex;

                          }

                      }

          

                      if (lastEx != null)

                          throw lastEx;

                  }

          

                  #endregion

              }

          }

This file has been truncated. show original

And the generation is done from here:

github.com

duplicati/duplicati/blob/master/Duplicati/Library/Modules/Builtin/RunScript.cs#L119


      
          

              m_logstorage = new FileBackedStringList();

              m_logscope = Logging.Log.StartScope(m => m_logstorage.Add(m.AsString(true)), m => {

          

                  if (filter.Matches(m.FilterTag, out var result, out var match))

                      return result;

                  else if (m.Level < logLevel)

                      return false;

          

                  return true;

              });

          }

          

          public string Key { get { return "runscript"; } }

          public string DisplayName { get { return Strings.RunScript.DisplayName; } }

          public string Description { get { return Strings.RunScript.Description; } }

          public bool LoadAsDefault  { get { return true; } }

          

          public IList<Duplicati.Library.Interface.ICommandLineArgument> SupportedCommands

          {

              get

You should be able to just do NewtonSoft.JsonConvert.SerializeObject(result) instead: