Announcing dupReport - A Duplicati Email Report Summary Generator

1

I’m happy to announce that I have finished development on version 2 of dupReport, a program that collects Duplicati’s backup status emails and creates a nice summary email like the one CrashPlan used to send.*

Version 1 of dupReport was released several weeks ago as a combination of bash and awk scripts for Linux. Admittedly, it was awful and ugly - but it worked! Version 2 has been completely rewritten in Python (Version 3) to be more portable and easier to use on multiple platforms. It’s especially helpful if you are running multiple Duplicati backup jobs on multiple source systems and destination locations.

Some of the features include:

  • Runs on multiple operating systems. (Tested so far on Linux (Debian 8) and Windows 10)
  • IMAP and POP3 support
  • SSL/TLS support for incoming/outgoing email transmissions.
  • Outgoing email supports both text and HTML format
  • All libraries are self-contained within the program. No need to install any additional packages (except Python3, of course)
  • No limit to the number of different backup jobs it can track
  • No limit to the frequency of job runs that it can track

Here is a sample report from the tool:

image
image.png754×704 59.5 KB

I welcome everyone to download the tool and try it on their system. I am also hoping that people can help test the program on platforms and configurations other than the limited few I have available. Specifically, I’m looking for feedback on the following:

  • Other OS platforms (MacOS, older Windows, other Linuxes, etc.)
  • Other email services (only tested with Gmail so far)
  • Email services that don’t require SSL/TLS (Gmail requires encryption for all connections)
  • Testing of command line and configuration options
  • Bug reports
  • Suggestions for future features & options

Basically, any feedback you can provide would be greatly appreciated. I’ve put a lot of effort into this and it works well for what I need, but I’m hoping that it can also work well for others looking for a similar tool.

Web site with a full description of the program can be found here.

Download the Python code from GitHub.

Discussion and feedback can be posted in this thread. You can also follow dupReport on Twitter @dupreport

Enjoy!

(*dupReport is an independent software project and is not affiliated with Duplicati or any of its developers.)

18 Likes
2

So it logs into GMail to scan for new reports?

How about just looking at a local mail client, say, Thunderbird?

3

Unfortunately, there is no way to know (1) what local client might be available on a given system, or (2) if there is an API or scripting method to use to get the client to extract the information we need. I tried to make the program as portable as possible so I opted for keeping all the email communications internal to the program.

If the concern is that you need to have the program log into your mail account (thus, presumably, introducing a security risk) you can always set up a separate email account used exclusively for Duplicati report emails. That way you can keep the report activity away from your main account and still have a place to collect all the emails.

Hopefully this addresses your question.

4

Fair enough. Debatable whether being specific to a client software vs a given webmail provider is more or less restrictive :slight_smile:

but I can make GMail work

5

Ah, I see your issue. dupReport is not specific to Gmail. It should work with any email provider that uses IMAP or POP3. However, I have only tested it with Gmail. I’m interested in getting feedback on its use with other providers as well.

6

So far the only issues I have seen with very small test is I can’t change the srcdestdelimiter setting. I have tried _ @ # all cause errors. In fact if it tries to parse any message with the @ in the subject errors are produced.

Since this was my own mailserver I just filtered the mails from Duplicati into its own folder and had dupreport use that.

This is Ubuntu 16.04 testing using dovecot and postfix

7

Hmm. Can you post a sample email subject and the three src/delim/dest specs from the dupReport.rc file you were using in your test? I can try to reproduce the issue here (and hopefully find a fix).

8

Had a chance to look at the @ in subject again. I think it might be unique to some mail clients. Sogo mail was creating the issue but it would accept same message from horde and roundcube webmail clients.

For example message with the subject of "this is a message with @ in subject’’

Sogo headers would look like.

User-Agent: SOGoMail 3.2.10
MIME-Version: 1.0
Date: Mon, 23 Oct 2017 15:04:46 -0400
Subject: this is a message with =?utf-8?q?=40?= in subject
Message-ID: <476-59ee3d80-b-1881d5c0@267997335>

The other two clients leave it alone

Subject: this is a message with @ in subject

Errors that happen

Traceback (most recent call last):
  File "./dupReport.py", line 856, in <module>
    process_mailbox_imap(mailBox)
  File "./dupReport.py", line 734, in process_mailbox_imap
    mParts = process_message(msg)                # Process message into parts
  File "./dupReport.py", line 480, in process_message
    msgParts['sourceComp'] = re.search(srcRegex, msgParts['subject']).group().split('-')[0]
AttributeError: 'NoneType' object has no attribute 'group'
9

Looking at the code it looks like the “-” is hard coded on line 478 and 479.

Shouldn’t that be something like

    msgParts['sourceComp'] = re.search(srcRegex, msgParts['subject']).group().split(options['srcdestdelimiter'])[0]
    msgParts['destComp'] = re.search(destRegex, msgParts['subject']).group().split(options['srcdestdelimiter'])[1]
1 Like
10

Looks nice!

This will track the last time a backup was run? That’s my biggest fear at this point with my current setup. I only get emails if there is a failure or other error. If the backup doesn’t run at all for some reason, I won’t know it.

Does it keep some stats stored locally or does it have to parse all backup report emails each time it runs?

11

This is why i like other people to look at my code. :grinning:

You actually uncovered 3 bugs on the code. The first you already guessed: the program was not properly using the srcdestdelimiter specification form the .rc file. Easy fix (mostly because you showed me where to look). But, your description of the problem led to the discovery of bug 2: if the subject field is matched but the delimiter is not found (because it was misspecified in the RC file). In this case dupReport will now skip the message, log the problem, and move on. But that got me thinking about the srcdestdelimiter specification itself. What if someone specifies a regex special character as the delimiter? Bug #3! Also fixed.

Anyway, version 2.0.2 now uploaded to GitHub and ready for more beatings. You can see the full description of the problem and fix in Issue #5. Thanks for bring this to my attention. I hope the rest of the testing goes smoothly.

12

drwtsn32,

The program will track the last time you receive an email from a backup, which is presumably when it completes. While this is not exactly what you asked about, the report does show the last time each backup was seen:

image

So, if the number of days starts creeping up you can check the backup program to see what the problem may be. Hopefully, that will give you what you’re looking for.

1 Like
13

Found another problem.

If dupreport tries to parse a failed upload it gives errors.

Temporary workaround was to add advance option in duplicatii

--send-mail-subject=Duplicati %OPERATIONNAME% (%PARSEDRESULT%) report for %backup-name%

Then edit the dupReport.rc file changing the subjectregex setting

subjectregex = ^Duplicati Backup \(Success\) report for

DupReport now should only process successful backups. You may to need to look at some kind of option to notify users when dupreport encounter a failed or warning status for %PARSEDRESULT%

14

Interesting. dR should either (1) not match the subject line and discard the message, or (2) match the subject line and pick up the failed status from the “Status:” field in the email. Would it be possible to post an email which causes the error?

P.S. Nice work-around :wink: Good to see the .rc options work as planned (usually).

15

Duplicati left the subject alone on the failed backup. This was an aborted backup so it was an expected error!

Duplicati sent the email in the same form as a successful backup just the body of message is what killed duplicati.

Subject remained “Duplicati Backup report for Test@B2” body contained the usual error stuff

Failed: Thread was being aborted.
Details: System.Threading.ThreadAbortException: Thread was being aborted.
   at System.Threading.WaitHandle.WaitOneNative(SafeHandle waitableSafeHandle, UInt32 millisecondsTimeout, Boolean hasThreadAffinity, Boolean exitContext)
   at System.Threading.WaitHandle.InternalWaitOne(SafeHandle waitableSafeHandle, Int64 millisecondsTimeout, Boolean hasThreadAffinity, Boolean exitContext)
   at System.Threading.WaitHandle.WaitOne(Int32 millisecondsTimeout, Boolean exitContext)
   at Duplicati.Library.Utility.BlockingQueue`1.Enqueue(T item)
   at Duplicati.Library.Main.BackendManager.WaitForEmpty(LocalDatabase db, IDbTransaction transation)
   at Duplicati.Library.Main.Operation.BackupHandler.Run(String[] sources, IFilter filter)
   at Duplicati.Library.Main.Controller.<>c__DisplayClass16_0.<Backup>b__0(BackupResults result)
   at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action`1 method)

Hence dupreport probably also needs some kind of error handling with %PARSEDRESULT% being processed in the subject line. If nothing more than the timestamp and failed in the report.

The %PARSEDRESULT% has possible values are Error, Warning, Success

Edit just forced another aborted upload, seems %PARSEDRESULT% also has “Fatal”

edit2:

After thinking about this some more I think subjectregex should be something like

subjectregex = ^Duplicati Backup \((Success|Error|Warn)\) report for

Assuming duplicati creates the proper output with Error and Warn dR will process the ParsedResult: in the email correctly. Not sure how to force Duplicati to create a warning/error message now that I need one.

Something separate to handle the Fatal error that %PARSEDRESULT% = Fatal maybe all that is needed

16

I’ve gotten a lot of great feedback and some bug reports from the initial release. Just wanted to let everyone know that there’s a new beta branch of the latest software (2.0.3) available on my GitHub site available for testing. Read the ‘changelog’ file for a list of the changes. The README file has all the latest command line and configuration options.

For those who have downloaded earlier releases, the software will update the dupReport.rc file automatically with new options. You may have the run the program twice; once to upgrade the options, the second time to run the report.

Please keep the suggestions and bug reports coming in. You can log issues on the GitHub site or just post them here.

Thanks for your support!

1 Like
17

I want to start by thanking you for writing this script – it is fantastic!

I did run into an issue that I have been able to work around – I’ve opened an issue on github for it.

Thank you once again for creating this tool – it is incredibly helpful!

Marc

18

Marc,

I posted a reply to your bug on GitHub. I think I know what’s going on, but I need some more debug code to be sure.

19

I published a forum post on how to configure Duplicati for gmail.

Part of that “how to” modifies the default email “subject” line to include the operation status. (“Duplicati %PARSEDRESULT%, %OPERATIONNAME% report for %backup-name%”. ) I did this because I like having the status in the subject line.

Unfortunately, this subject line will not match the default regex used by dupReport.

Modifying the “subjectregex =” line in dupReport.rc as shown below will ensure that dupReport.py work for either the Duplicati default subject line or the subject line used in my writeup:

subjectregex = ^Duplicati ([\w ]*, |)Backup report for

I’ve included a note about this in my writeup and providing it here in case others find it useful…

Marc

1 Like
20

dcurrey had a similar thought about using %PARSEDRESULT% a few posts back. His regex ended up being:

subjectregex = ^Duplicati Backup ((Success|Error|Warn)) report for

Either will work with dupReport, as long as the regex is properly specified and matches the --send-mail-subject option in Duplicati.

Thanks for posting this!