Announcing dupReport - A Duplicati Email Report Summary Generator

Released 2.2.2 into the wild today. Found a couple of bugs during testing but it seems pretty stable now. All caught up on the bug/feature backlog. Let me know if there’s anything else you’d like to see.

HG

1 Like

Hey handyguy. Thanks for all the work! I am new to all this…when you say this works with an email server, what does that mean? I have a plain personal gmail account. Will that work or do I need something else?

Hello whitenack
Your gmail account can be theoretically used, but non one will you recommend that.
Password to your account must be stored in readable text in configure file. So it is huge security risk for you.

But you can create new gmail account only for this purpose and use this account in dupReport - this will be best scenario.

Sorry, @whitenack, I’ve been traveling for the past couple of weeks and not paying attention to email. @mr-flibble is correct in that you should consider using a separate account for your Duplicati & dupReport emails. Not only will it address the security concerns of needing to store your email password in the .rc file, it also keeps your main account from clogging up with a lot of backup-related emails.

I’ve been thinking of a more secure way of storing the email password for dupReport on and off for a while, but every method I work on just gets too complicated for the tool. If someone else can think of a good, secure, easy way of storing the password feel free to chime in.

I want to let everyone know that dupReport version 2.2.3 has been uploaded to the pre_prod branch on GitHub. The biggest feature change is support for the Apprise push notification service. The ReadMe has been updated with instructions on how to use Apprise from within dupReport.

I’m interested in getting feedback on the new feature, so please check it out when you get a chance and let me know your thoughts.

Thanks,

HG

dupReport 2.2.3 has been released into the wild. It adds support for the Apprise notification service as well as some minor bug fixes. If you’re not interested in using Apprise there’s no need to upgrade immediately (but being on the latest code is always nice :grinning:).

HG

I missed it by 2 days: dupReport is one year old this week and going strong. Thanks to everyone who has downloaded and tried the program, and also thanks to those who have sent in suggestions and comments (good and bad). I don’t have download counts (and there aren’t any trackers in the code :wink: ), but hopefully people have found it useful! :birthday:

2 Likes

Hi handyguy,

I’m having a problem with running dupReport.py multiple times in one session. For instance, I have a bash script set up that does the following:

/dupReport/testrun.sh
#!/bin/bash
/dupReport/dupReport.py -r /dupReport/rc/location1
/dupReport/dupReport.py -r /dupReport/rc/location2
/dupReport/dupReport.py -r /dupReport/rc/location3
/dupReport/dupReport.py -r /dupReport/rc/location4
/dupReport/dupReport.py -r /dupReport/rc/location5

… and a crontab file that looks like this:
0 07 * * 1-5 /dupReport/dupReport.py -r /dupReport/rc/location1
0 13 * * 1-5 /dupReport/dupReport.py -r /dupReport/rc/location2
0 13 * * 1-5 /dupReport/dupReport.py -r /dupReport/rc/location3
0 13 * * 1-5 /dupReport/dupReport.py -r /dupReport/rc/location4
0 13 * * 1-5 /dupReport/dupReport.py -r /dupReport/rc/location5

The script (above) is used for me to test what backup reports look like before they are put into production and end-users get the reports.

The crontab (below) is how I’d like to run them once they are in production.

Thankfully, I’m the only one getting the emails right now. Otherwise, I’d have a really big problem.

The output of both scenarios above generates 5 emails.

The first email looks perfect.
The second email contains the data for the first location and then the data for the second.
The third email contains the data for the first through third locations.
The fourth email contains the data for the first through fourth locations
The fifth email contains the data for the first through fifth locations
(… and a partridge in a pear treeeee)

So, it’s obvious that the do-while or for-x-to-y statement is running an append on a variable in the script that contains the output. The problem is, at the end of each script, the variables don’t appear to be getting cleared.

Is this an easy fix?

Thanks!
-Paul

Paul,

Can you please open up an issue on the GitHub site? That way we can converse & figure this out without clogging up the forum here. Thanks

Done, and thanks in advance!

Version 2.2.4 (Beta 2) of dupReport is now available for testing in the pre_prod branch on GitHub. The main change in this release is the addition of a "Duration" field on the report to show how long the backup job ran. See the readme file for information on how this works.

WARNING. The database structure was changed slightly for this release, so please back up your .RC and .DB files before running this version (always a good practice for any upgrade) and run the program with the -v3 option (for very verbose logging) to catch any errors that may crop up.

Thanks to @brogel and @mr-flibble for help with this release.

Enjoy the new code! Please let me know about any issues (and successes :slight_smile:)

HG

1 Like

Version 2.2.4 has now been fully release into the “master” branch. Enjoy! :slight_smile:

Hello,

I’m trying to setup the dupReport, but i’m stuck with 2 problems.
First, i want to setup a pop3 transport (imap is not an option for me).
I’ve tried the latest release (2.2.4), even tried the “Issue_103” release, but i get the error below.
It seems that python 3, doesn’t have encode/decode anymore and i have tried many versions o python 3.
The problem seems to begin with Release_2.2.0 because with the Release_2.1.0 there is no such error.

python.exe "F:\dupReport-Issue_103\dupReport.py" -v3
Connecting to email servers.
Analyzing email messages.
Traceback (most recent call last):
  File "F:\dupReport-Issue_103\dupReport.py", line 234, in <module>
    nxtMsg = globs.inServer.processNextMessage()
  File "F:\dupReport-Issue_103\dremail.py", line 295, in processNextMessage
    msgParts['date'], msgParts['subject'], msgParts['messageId'] = self.extractHeaders(body.decode('utf-8'))
AttributeError: 'list' object has no attribute 'decode'

Second, while trying Release_2.1.0, the email report is generated and send to me, but no duplicati emails are processed.
I haven’t changed the “send-mail-subject”, so i left the subjectregex as it is (^Duplicati Backup report for).
I’ve attached the dupReport.log (without the personal info).

I would be thankful for any help

dupReport.zip (2.7 KB)

@peoman, I’ll start researching this. In the meantime, please open up an issue on the GitHub site. We can track progress there without cluttering up this forum.

Also, please re-run the 2.2.4 version (the one that’s giving you the error messages) with the -v3 option to get the full debugging log file and post the log file to the issue on GitHub.

I suspect that the 2.1.0 issue you are seeing is because dupReport isn’t finding any “emails of interest” to report on. This most often happens because of performing multiple program runs on the same inbox sequentially. dR remembers what emails it’s already seen and won’t report on them a second time. If you suspect this may be the case, re-run the program using the ‘-b’ option to roll back the database to a previous date/time to see if that changes the result. If you can also do that run with the -v3 option and post the log file that would be really helpful as well.

@peoman, I was able to confirm the first bug/crash with extractHeaders(), so no need to upload log files for that. The problem is with the POP3 header processing, not with the Python3 libraries, so I should be able to detrmine a fix (fingers crossed! :slight_smile: ) Clearly POP3 is not a popular protocol with dupReport users, since this bug appears to have existed for a while and you’re the first to report it!

I opened up an issue for you here (Issue 105). Check on the GitHub site for progress or fixes.

HG

I don’t think that the emails are already seen from dupReport.
I’ve tried after new mails arrived but still no luck.
I’ve even tried after “reinstalling” the dupReport (extracted on a different location and recreated rc and db)

The dupReport.zip I’ve attached to my previous reply, is from 2.1.0 with the -v3 option.

Thanks

@peoman, what email service are you using? Is it Gmail or some other service?

Microsoft Exchange Server 2010

@peoman

Some good news and some bad news. The good news is that your report led to finding a whole lot of bugs in some sadly-neglected parts of the POP3 handling code, including the crash error that you initially reported. I believe all of those have now been fixed and uploaded to the Issue_105 branch on Github. Please download the code from this branch and see if it fixes your crashing issue.

The bad news is that the lack of line items in your report doesn’t seem to be a bug, rather it’s a bad artifact of using POP3. POP3 seems to work slightly differently on different implementations, but in general it automatically marks any emails that are read as “seen” and only lets you download them once. After that, it won’t let you download them again even if they are still there on the server. That’s why you were able to generate a report the first time but it came up empty after that. It also means that if you have a client that reads mail from your inbox before dR reads them, dR will never see any new mail because your client has already had them marked as read.

One way to get around this might be to use a “Leave messages on the server” option. This option tells the server to leave the emails marked as unread even if they are downloaded to a client. Support for this and the method for enabling it seems to vary from system to system, but instructions for doing this on Exchange/Outlook are here. If you’re not using Outlook hopefully this will give you enough information to get it working on your client.

I am using Gmail as my server and was able to reproduce both the conditions resulting an empty report (what you originally reported) and a successful run using the “Leave on server” process. Gmail allows you to set this on the server, whereas I believe Exchange requires you to set this on the client (make sure you set it for all the clients you use for this account, or it won’t work). As an added evil twist, Gmail also requires you to put a “recent:” tag in front of your user name in order to fetch the most recent emails in your inbox instead of the oldest. (Like I said, POP3 implementations vary from system to system.) I don’t think you’ll have that issue on Exchange, but since I don’t have an Exchange server to test with I wasn’t able to reproduce/verify that piece.

Anyway, please download the Issue_105 code and see if you can get it working on your system. If it checks out I’ll post it to pre_prod for others to test (there are some other enhancements I included as well). Let me know how it goes. To keep this thread uncluttered, please post any bug reports for this on the Issue_108 thread on GitHub. (My numbering scheme needs some improvement! :slight_smile: )

Good luck,

HG