Excessive failures/warnings - AKA making Duplicati more stable

1

Hi all,

I have Duplicati running in datacenter environments scattered across about 20 sites. I’ve been (and remain!) a fan of the project for perhaps a year now and I’ve been slowing rolling it out to more and more servers and workstations that need the backups.

I’m also using all the notification tools available – I’ve got duplicati-monitoring.com in use on all of my backup jobs so I have one dashboard to look at. Then I’ve also got dupReports running for each location, with the daily reports going to the local network admin of that site and also the management of that site.

Here’s what I’m noticing from an enterprise administration point of view: The amount of jobs that “fail” is abnormally large. For example, I’m looking at my Duplicati Monitoring report for this morning, and the subject line states “34 backup sets with errors, 15 with warnings, 31 OK”… so, out of 80 backup jobs I have set up, 34 fail out, 15 succeed with warnings, and 31 are OK. That means that less than 50% of my backups are successful. I’m utilizing snapshot-policy=auto where feasible and snapshot-policy=required where absolutely necessary. I’m excluding windows system files, temp files, etc. where possible as well.

Drilling down into this data a little deeper, I start looking at the reasons the backups are failing, and some of them are due to storage devices being offline – that’s fair, I can address those individually – but a lot of them are due to differences in the local vs remote databases, or due to “fileset version” errors. Examples:

Failed: Unexpected difference in fileset version 5: 4/7/2019 8:00:01 PM (database id: 161), found 261351 entries, but expected 261352

Failed: Found remote files reported as duplicates, either the backend module is broken or you need to manually remove the extra copies.

I’ve been scouring through my email and I cannot find a good example of it, but I also get a lot of errors where the local database and the remote database don’t match up, meaning I end up trying a repair and ultimately deleting the database on both ends and then letting Duplicati recreate the database.

Overall, all of the above are just examples. My big question is this: What can be done to make Duplicati a “more stable” backup solution? Right now I get emails from managers who do not care what the detail message of the failure says, they just care that the backup notice says “fail” or “success”, and they get a little testy about “warning” – so is there anything that can be done on the Duplicati side to identify when a database does not match and “auto-magically” rebuild it? Is there any way to take some of these errors and resolve them so human intervention is not required to fix the problem?

Thanks in advance!

1 Like
2

These exact type of problems have meant I have stopped using Duplicati other than on an ongoing test basis. I do hope the problems get resolved, but until they do I cannot rely on it.

2 Likes
3

Hello @anomaly0617

“Unexpected difference in fileset” test case and code clue #3800 has a fix for at least one way it happens in
Release: 2.0.4.22 (canary) 2019-06-30 which I would encourage you to try on a test system if you have one that’s known to hit the error often. Canary is bleeding-edge, but this one is Release: 2.0.4.21 (experimental) 2019-06-28 plus limited other fixes, and “experimental” is the final test before beta, so testing will help beta.

Testing helps, and testers with enough systems and expertise to file useful issues (ideally with information someone could use to reproduce or understand the issue) can offer big benefits to knowing where to start.

The rate of development is limited by the time, skills, and facilities that development volunteers can donate. Sometimes users have lots more equipment and variety, and the trick is to get feedback in a useful format.

There are very few backends that are able to achieve this feat, if it’s real – have you checked the backend, and is it Google Drive by any chance? google drive API allows duplicates? Is there an issue or forum post? There weren’t any I could find with adequate posted information such as lines from the –log-file option with –log-file-log-level=retry (or higher if one is trying finest-grained debugging at the expense of very large logs, however it becomes so large that that’s more something a developer would do, given steps to reproduce).

In terms of new features and abilities, discussion on those would perhaps be better as individual topics in Features category. That doesn’t bring resources, though, and there are a lot of open issues and requests.

Specifically on recreate or repair though, that’s a sore spot at the moment even when run manually, so an automatic run might be controversial. There was a rewrite started, but I’m not sure when/if it will come out.

4

I started it quite some time ago, but I have not had the time to complete it. It would require a few days of work to complete the rewrite, but it does not look I will get that any time soon :frowning: