Announcing dupReport - A Duplicati Email Report Summary Generator

Interesting. dR should either (1) not match the subject line and discard the message, or (2) match the subject line and pick up the failed status from the “Status:” field in the email. Would it be possible to post an email which causes the error?

P.S. Nice work-around :wink: Good to see the .rc options work as planned (usually).

Duplicati left the subject alone on the failed backup. This was an aborted backup so it was an expected error!

Duplicati sent the email in the same form as a successful backup just the body of message is what killed duplicati.

Subject remained “Duplicati Backup report for Test@B2” body contained the usual error stuff

Failed: Thread was being aborted.
Details: System.Threading.ThreadAbortException: Thread was being aborted.
   at System.Threading.WaitHandle.WaitOneNative(SafeHandle waitableSafeHandle, UInt32 millisecondsTimeout, Boolean hasThreadAffinity, Boolean exitContext)
   at System.Threading.WaitHandle.InternalWaitOne(SafeHandle waitableSafeHandle, Int64 millisecondsTimeout, Boolean hasThreadAffinity, Boolean exitContext)
   at System.Threading.WaitHandle.WaitOne(Int32 millisecondsTimeout, Boolean exitContext)
   at Duplicati.Library.Utility.BlockingQueue`1.Enqueue(T item)
   at Duplicati.Library.Main.BackendManager.WaitForEmpty(LocalDatabase db, IDbTransaction transation)
   at Duplicati.Library.Main.Operation.BackupHandler.Run(String[] sources, IFilter filter)
   at Duplicati.Library.Main.Controller.<>c__DisplayClass16_0.<Backup>b__0(BackupResults result)
   at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action`1 method)

Hence dupreport probably also needs some kind of error handling with %PARSEDRESULT% being processed in the subject line. If nothing more than the timestamp and failed in the report.

The %PARSEDRESULT% has possible values are Error, Warning, Success

Edit just forced another aborted upload, seems %PARSEDRESULT% also has “Fatal”

edit2:

After thinking about this some more I think subjectregex should be something like

subjectregex = ^Duplicati Backup \((Success|Error|Warn)\) report for

Assuming duplicati creates the proper output with Error and Warn dR will process the ParsedResult: in the email correctly. Not sure how to force Duplicati to create a warning/error message now that I need one.

Something separate to handle the Fatal error that %PARSEDRESULT% = Fatal maybe all that is needed

I’ve gotten a lot of great feedback and some bug reports from the initial release. Just wanted to let everyone know that there’s a new beta branch of the latest software (2.0.3) available on my GitHub site available for testing. Read the ‘changelog’ file for a list of the changes. The README file has all the latest command line and configuration options.

For those who have downloaded earlier releases, the software will update the dupReport.rc file automatically with new options. You may have the run the program twice; once to upgrade the options, the second time to run the report.

Please keep the suggestions and bug reports coming in. You can log issues on the GitHub site or just post them here.

Thanks for your support!

1 Like

I want to start by thanking you for writing this script – it is fantastic!

I did run into an issue that I have been able to work around – I’ve opened an issue on github for it.

Thank you once again for creating this tool – it is incredibly helpful!

Marc

Marc,

I posted a reply to your bug on GitHub. I think I know what’s going on, but I need some more debug code to be sure.

I published a forum post on how to configure Duplicati for gmail.

Part of that “how to” modifies the default email “subject” line to include the operation status. (“Duplicati %PARSEDRESULT%, %OPERATIONNAME% report for %backup-name%”. ) I did this because I like having the status in the subject line.

Unfortunately, this subject line will not match the default regex used by dupReport.

Modifying the “subjectregex =” line in dupReport.rc as shown below will ensure that dupReport.py work for either the Duplicati default subject line or the subject line used in my writeup:

subjectregex = ^Duplicati ([\w ]*, |)Backup report for

I’ve included a note about this in my writeup and providing it here in case others find it useful…

Marc

1 Like

dcurrey had a similar thought about using %PARSEDRESULT% a few posts back. His regex ended up being:

subjectregex = ^Duplicati Backup ((Success|Error|Warn)) report for

Either will work with dupReport, as long as the regex is properly specified and matches the --send-mail-subject option in Duplicati.

Thanks for posting this!

Just to let everyone know, a new version of dupReport beta 2.0.3 has been posted to the beta branch on the dupReport project on GitHub. Lots of bug fixes and a few new options. Check it out if you can.

I’ll let it ‘bake in’ for a week or so and if there are no more problems I’ll merge it into the master branch sometime next week.

Enjoy!

I have a suggestion for an alternative way of formatting reports. The basic idea is to create a visual grouping when the same source is backed up to multiple destinations.

A picture is worth a thousand words – this google sheet is a mockup of what I am thinking.

Food for thought…

Marc

1 Like

Hmmm. Interesting approach. It’s totally doable as the information is already in the database, and is basically just a variation of the way the report is generated now. I have a feature on the drawing board that would allow the user to select the columns displayed on the report. It may be time to re-think the way the reports are produced. Let me think about this for a bit…

Fantastic idea – I love it!

Another idea for enabling report customization that I think would pair well with what you described: Allow the user to format the sub-header row that currently contains “”*** source-name to destination-name ***" with a template that allows keyword substitution. ie, Something like %SOURCENAME% would be substituted with the actual source name.

A new sub-head row would need to be generated anytime one of the keywords used in the template changes.

Marc

1 Like

Marc,

I coded up a rough demo version of what you’re talking about in the source/destination grouping, just as a proof-of-concept. It works, but it’s far from elegant. Or pretty. I still need to think about how to best handle the sub-header formatting.

Right now I’m trying to keep 2.0.3 stable for a week or so then I can merge it into the master branch as an official release. As soon as that’s done I can open up 2.0.4 for development and start on the reporting fixes (as well as some other goodies).

Stay tuned…

Very cool! Let me know if you want me to do some testing…

Marc

Hey folks!

dupReport 2.0.3 has been released on GitHub. Lots of bug fixes and a few extra features. Check out the changelog file for details. If you haven’t upgraded since the early days, you might want to grab this release.

I’m now turning my attention to 2.0.4, which will be mostly related to improvements in the reporting features of the program. A lot of it is based on some suggestions by Marc Aronson about different ways to sort the report, but I’m also looking at a couple of more features, lots of internal code improvement, and (gasp!) bug fixes. I’m also moving my development platform from Linux to Windows. The program is getting too big for nano editing and command-line git.

This one may take a while, but I’ll keep everyone updated on progress.

Enjoy!

1 Like

Hi - just downloaded dupReport 2.0.3 to give it a try.

The initialization step using dupReport.py -i causes an exception: NameError: name ‘write_log_message’ is not defined.

My assumption is that it should be ‘write_log_entry’ which seems to work fine.

1 Like

Not sure how that happened. I had a bit of trouble switching development platforms last night and it may have crept into there. Just uploaded fixed code to the master branch. Thanks for the notice.

Due to popular demand (a couple of people asked), I have opened up a Twitter account for dupReport. Follow @dupreport for release announcements, support, and program info. Low volume but high quality :wink: .

1 Like

Hello HandyGuy, I made a small change, see if you find it interesting: a highlight in the lines with the names of the jobs. When you have a lot of small backups, I found the report look a bit confusing, so I put a background in the line with the names. Look:

new

I just put the color tag here (line 690):

msgHtml = msgHtml + '<td bgcolor="#E6E6E6" align="center" colspan = "11"><b>{}</b></td>'.format(txt[0])
1 Like

Interesting option. I’ll add it to the 2.1 features list.

Hey, everyone: version 2.0.4 has been released and is now available on the GitHub ‘master’ branch. Some bug fixes in this release, including better handling for all sorts of date & time formats. Check it out!

2 Likes