Some good news and some bad news. The good news is that your report led to finding a whole lot of bugs in some sadly-neglected parts of the POP3 handling code, including the crash error that you initially reported. I believe all of those have now been fixed and uploaded to the Issue_105 branch on Github. Please download the code from this branch and see if it fixes your crashing issue.
The bad news is that the lack of line items in your report doesn’t seem to be a bug, rather it’s a bad artifact of using POP3. POP3 seems to work slightly differently on different implementations, but in general it automatically marks any emails that are read as “seen” and only lets you download them once. After that, it won’t let you download them again even if they are still there on the server. That’s why you were able to generate a report the first time but it came up empty after that. It also means that if you have a client that reads mail from your inbox before dR reads them, dR will never see any new mail because your client has already had them marked as read.
One way to get around this might be to use a “Leave messages on the server” option. This option tells the server to leave the emails marked as unread even if they are downloaded to a client. Support for this and the method for enabling it seems to vary from system to system, but instructions for doing this on Exchange/Outlook are here. If you’re not using Outlook hopefully this will give you enough information to get it working on your client.
I am using Gmail as my server and was able to reproduce both the conditions resulting an empty report (what you originally reported) and a successful run using the “Leave on server” process. Gmail allows you to set this on the server, whereas I believe Exchange requires you to set this on the client (make sure you set it for all the clients you use for this account, or it won’t work). As an added evil twist, Gmail also requires you to put a “recent:” tag in front of your user name in order to fetch the most recent emails in your inbox instead of the oldest. (Like I said, POP3 implementations vary from system to system.) I don’t think you’ll have that issue on Exchange, but since I don’t have an Exchange server to test with I wasn’t able to reproduce/verify that piece.
Anyway, please download the Issue_105 code and see if you can get it working on your system. If it checks out I’ll post it to pre_prod for others to test (there are some other enhancements I included as well). Let me know how it goes. To keep this thread uncluttered, please post any bug reports for this on the Issue_108 thread on GitHub. (My numbering scheme needs some improvement! )
I’m happy to report that dupReport Version 2.2.5 is available in the pre_prod branch on GitHub. In addition to fixing some latent POP3 bugs, it adds the ability to mask sensitive data (by default) such as user name, password, email server, etc in the log files. This comes in handy if you have to send the log files somewhere else. Check out the README.md in 2.2.5 for more details.
Hi, thank-you for creating this, definitely missed its information when I moved away from CrashPlan. I have it running nice on a small Fedora box I have for such things.
One question: can it mark the emails as “read” once processed? I use IMAP on my own mail server and can see the dedicated mailbox on my normal client, but all the emails remain unread. So unless I missed a setting it would be nice if dupReport marked each as read once it had processed them. Also simple way to know that all is still well.
@Taomyn, the program takes a “leave no trace” approach when accessing mailboxes, as I can’t be sure what other programs may be using the same mailbox and I don’t want to interfere with their operation. However, I could probably add an option to allow for marking emails as read once dR sees them. Not sure about POP3 but pretty sure it can be done for IMAP.
Please open up an issue on the GitHub site and I’ll take a look into it. It may be a couple of weeks before I can get to it, as I have a few other things piled up in front of it.
Hey, all. dupReport veriosn 2.2.6 is now available for testing on the pre_prod branch on GitHub. This release incorporates @Taomyn’s suggestion of enabling the ability to mark messages as read/seen once dupReport has finished processing them. Please download and take it for a test spin if you are so inclined.
Problem with that is the script would need access to every single database for each individual job on each individual machine you’re running Duplicati on. The emails being sent are generally all in a central location.
Changing dupReport.rc ->incoming->inbox to other email box gives me this result:
C:\dupReport-master>c:\python37\python dupReport.py
Traceback (most recent call last):
File “dupReport.py”, line 232, in
newMessages = globs.inServer.checkForMessages()
File “C:\dupReport-master\dremail.py”, line 206, in checkForMessages
retVal, data = self.server.search(None, “ALL”)
File “c:\python37\lib\imaplib.py”, line 723, in search
typ, dat = self._simple_command(name, *criteria)
File “c:\python37\lib\imaplib.py”, line 1196, in _simple_command
return self._command_complete(name, self._command(name, *args))
File “c:\python37\lib\imaplib.py”, line 944, in _command
', '.join(Commands[name])))
imaplib.error: command SEARCH illegal in state AUTH, only allowed in states SELECTED
@axiomcs, please open up an issue on the dupReport GitHub site for this issue. Also paste in the issue the settings from the [incoming] section of the .rc file (without IDs or passwords - I don’t need those).
Properly parse “Log data” field, newly introduced in Duplicati circa November 2018
Parse JSON-formatted emails from Duplicati job runs
Added a “Duplicati version” field option to report. Can be helpful in discovering when one of your systems is behind in Duplicati updates
Added -F option. Same as -f, but also sends resulting output file as an attachment to the email.
If you’re feeling adventurous please download the new code and check it out. I’ve been running it for about 3 weeks on both POP3 & IMAP and it appears to be pretty stable. NOTE: The new version updates the database format, so please back up your dupreport.db file before running for the first time. Let me know if you have any issues with the update.
I upgraded from 2.2.6 after running it one last time and backing up the entire folder.
I re-ran it manually and nothing bad has happened, just very uneventful report, and will wait for next run after some backups have run through.
Two questions:
1. How do I enable the “Duplicati version” option? Couldn’t see it in the run I made. Answered my own question, it was just empty I presume because it has still yet to process a new backup job
2. The .rc file still says “version=2.2.5” even though I was on 2.2.6 before and now on 2.2.7 - is this expected for the moment?
The “[main] version” option in the .rc file is used to determine whether the .rc file format needs to be upgraded. So, it really should be labeled ‘rcversion.’ If you look at the code for database management (db.py) it uses a similar technique. I haven’t changed the .rc format much since dupReport 2.1, so it hasn’t really mattered much.
If you want to see what version of the software you are running you can look at the bottom of the output report for a line that says something like:
Report generated by dupReport (https://github.com/HandyGuySoftware/dupReport) Version 2.2.7 (Beta 1)
or check the line at the top of the globs.py file that reads:
version=[x,x,x] # Program Version
status=\'Beta 1\'
However, your note got me looking into that section of the code (one of the earliest modules in the 2.x rewrite) and I noticed that what actually gets written into the [main] version option is the version of the program the first time you run dupReport. So, for example, on my test system the line reads:
version = 2.1.0
Judging from your message I am assuming that the first time you ran dupReport was on version 2.2.5. Again, since the format hasn’t changed much it won’t affect program operation, but it does explain what you are seeing.
I opened an issue for this on GitHub to make the .rc file version representation more accurate and will probably upload a revised beta when that gets fixed. You can track progress there or stay tuned here for the new beta announcement.
Well, that didn’t take long. Updated the code to bring all older .rc file versions up to a uniform 3.0.0, where it will probably stay in perpetuity. Also took the opportunity to update some of the old code while I was in there.
I upgraded successfully from the previous pre_prod and everything looks good = the .rc has the new version number.
One small request: could the Duplicati version also appear in the “Backup Sets Last Seen” table at the start? Would make it easier to check, plus when no backup log was processed for a machine it doesn’t appear in the main summary report table.
)