Announcing dupReport - A Duplicati Email Report Summary Generator

#62

Hi Folks!

dupReport Version 2.2.2 has been moved to the pre_prod branch on GitHub. This version adds a “report interval” option for managing & reporting backup jobs that run at intervals other than once a day. If a backup from a source/destination pair is not seen while scanning the emails but the number of days since the last backup is less than the backupinterval value, the program will simply print a notification message rather than the standard warning message. For example (from the ‘bydest’ report):

image
image.png747×66 4.33 KB

The first line represents a backup that missed its daily execution. The second line represents a backup that only runs every 5 days.

See the readme file for more details on how this works.

There were a couple of different ways to go with this feature, so I’m interested in having folks try it out & see if there is a better way to do it.

Enjoy!

HG

#63

Released 2.2.2 into the wild today. Found a couple of bugs during testing but it seems pretty stable now. All caught up on the bug/feature backlog. Let me know if there’s anything else you’d like to see.

HG

#64

Hey handyguy. Thanks for all the work! I am new to all this…when you say this works with an email server, what does that mean? I have a plain personal gmail account. Will that work or do I need something else?

#65

Hello whitenack
Your gmail account can be theoretically used, but non one will you recommend that.
Password to your account must be stored in readable text in configure file. So it is huge security risk for you.

But you can create new gmail account only for this purpose and use this account in dupReport - this will be best scenario.

#66

Sorry, @whitenack, I’ve been traveling for the past couple of weeks and not paying attention to email. @mr-flibble is correct in that you should consider using a separate account for your Duplicati & dupReport emails. Not only will it address the security concerns of needing to store your email password in the .rc file, it also keeps your main account from clogging up with a lot of backup-related emails.

I’ve been thinking of a more secure way of storing the email password for dupReport on and off for a while, but every method I work on just gets too complicated for the tool. If someone else can think of a good, secure, easy way of storing the password feel free to chime in.

#67

I want to let everyone know that dupReport version 2.2.3 has been uploaded to the pre_prod branch on GitHub. The biggest feature change is support for the Apprise push notification service. The ReadMe has been updated with instructions on how to use Apprise from within dupReport.

I’m interested in getting feedback on the new feature, so please check it out when you get a chance and let me know your thoughts.

Thanks,

HG

#68

dupReport 2.2.3 has been released into the wild. It adds support for the Apprise notification service as well as some minor bug fixes. If you’re not interested in using Apprise there’s no need to upgrade immediately (but being on the latest code is always nice :grinning:).

HG

#69

I missed it by 2 days: dupReport is one year old this week and going strong. Thanks to everyone who has downloaded and tried the program, and also thanks to those who have sent in suggestions and comments (good and bad). I don’t have download counts (and there aren’t any trackers in the code :wink: ), but hopefully people have found it useful! :birthday:

#70

Hi handyguy,

I’m having a problem with running dupReport.py multiple times in one session. For instance, I have a bash script set up that does the following:

/dupReport/testrun.sh
#!/bin/bash
/dupReport/dupReport.py -r /dupReport/rc/location1
/dupReport/dupReport.py -r /dupReport/rc/location2
/dupReport/dupReport.py -r /dupReport/rc/location3
/dupReport/dupReport.py -r /dupReport/rc/location4
/dupReport/dupReport.py -r /dupReport/rc/location5

… and a crontab file that looks like this:
0 07 * * 1-5 /dupReport/dupReport.py -r /dupReport/rc/location1
0 13 * * 1-5 /dupReport/dupReport.py -r /dupReport/rc/location2
0 13 * * 1-5 /dupReport/dupReport.py -r /dupReport/rc/location3
0 13 * * 1-5 /dupReport/dupReport.py -r /dupReport/rc/location4
0 13 * * 1-5 /dupReport/dupReport.py -r /dupReport/rc/location5

The script (above) is used for me to test what backup reports look like before they are put into production and end-users get the reports.

The crontab (below) is how I’d like to run them once they are in production.

Thankfully, I’m the only one getting the emails right now. Otherwise, I’d have a really big problem.

The output of both scenarios above generates 5 emails.

The first email looks perfect.
The second email contains the data for the first location and then the data for the second.
The third email contains the data for the first through third locations.
The fourth email contains the data for the first through fourth locations
The fifth email contains the data for the first through fifth locations
(… and a partridge in a pear treeeee)

So, it’s obvious that the do-while or for-x-to-y statement is running an append on a variable in the script that contains the output. The problem is, at the end of each script, the variables don’t appear to be getting cleared.

Is this an easy fix?

Thanks!
-Paul

#71

Paul,

Can you please open up an issue on the GitHub site? That way we can converse & figure this out without clogging up the forum here. Thanks

#72

Done, and thanks in advance!

#73

Version 2.2.4 (Beta 2) of dupReport is now available for testing in the pre_prod branch on GitHub. The main change in this release is the addition of a "Duration" field on the report to show how long the backup job ran. See the readme file for information on how this works.

WARNING. The database structure was changed slightly for this release, so please back up your .RC and .DB files before running this version (always a good practice for any upgrade) and run the program with the -v3 option (for very verbose logging) to catch any errors that may crop up.

Thanks to @brogel and @mr-flibble for help with this release.

Enjoy the new code! Please let me know about any issues (and successes :slight_smile:)

HG

#74

Version 2.2.4 has now been fully release into the “master” branch. Enjoy! :slight_smile:

#75

Hello,

I’m trying to setup the dupReport, but i’m stuck with 2 problems.
First, i want to setup a pop3 transport (imap is not an option for me).
I’ve tried the latest release (2.2.4), even tried the “Issue_103” release, but i get the error below.
It seems that python 3, doesn’t have encode/decode anymore and i have tried many versions o python 3.
The problem seems to begin with Release_2.2.0 because with the Release_2.1.0 there is no such error.

python.exe "F:\dupReport-Issue_103\dupReport.py" -v3
Connecting to email servers.
Analyzing email messages.
Traceback (most recent call last):
  File "F:\dupReport-Issue_103\dupReport.py", line 234, in <module>
    nxtMsg = globs.inServer.processNextMessage()
  File "F:\dupReport-Issue_103\dremail.py", line 295, in processNextMessage
    msgParts['date'], msgParts['subject'], msgParts['messageId'] = self.extractHeaders(body.decode('utf-8'))
AttributeError: 'list' object has no attribute 'decode'

Second, while trying Release_2.1.0, the email report is generated and send to me, but no duplicati emails are processed.
I haven’t changed the “send-mail-subject”, so i left the subjectregex as it is (^Duplicati Backup report for).
I’ve attached the dupReport.log (without the personal info).

I would be thankful for any help

dupReport.zip (2.7 KB)

#76

@peoman, I’ll start researching this. In the meantime, please open up an issue on the GitHub site. We can track progress there without cluttering up this forum.

#77

Also, please re-run the 2.2.4 version (the one that’s giving you the error messages) with the -v3 option to get the full debugging log file and post the log file to the issue on GitHub.

I suspect that the 2.1.0 issue you are seeing is because dupReport isn’t finding any “emails of interest” to report on. This most often happens because of performing multiple program runs on the same inbox sequentially. dR remembers what emails it’s already seen and won’t report on them a second time. If you suspect this may be the case, re-run the program using the ‘-b’ option to roll back the database to a previous date/time to see if that changes the result. If you can also do that run with the -v3 option and post the log file that would be really helpful as well.

#78

@peoman, I was able to confirm the first bug/crash with extractHeaders(), so no need to upload log files for that. The problem is with the POP3 header processing, not with the Python3 libraries, so I should be able to detrmine a fix (fingers crossed! :slight_smile: ) Clearly POP3 is not a popular protocol with dupReport users, since this bug appears to have existed for a while and you’re the first to report it!

I opened up an issue for you here (Issue 105). Check on the GitHub site for progress or fixes.

HG

#79

I don’t think that the emails are already seen from dupReport.
I’ve tried after new mails arrived but still no luck.
I’ve even tried after “reinstalling” the dupReport (extracted on a different location and recreated rc and db)

The dupReport.zip I’ve attached to my previous reply, is from 2.1.0 with the -v3 option.

Thanks

#80

@peoman, what email service are you using? Is it Gmail or some other service?

#81

Microsoft Exchange Server 2010