After update to latest beta, backups broken

macOS: After downloading and installing the offered beta 2.0.4.23_beta_2019-07-14 (via the web interface), the first run of my backup says: Unexpected difference in fileset version 11: 3/25/2019 9:28:51 AM (database id: 119), found 221452 entries, but expected 221453

Repair database gave ‘1 warning’ (not shown) and doesn’t help

What now?

After ‘Recreate’, I got:
The database was attempted repaired, but the repair did not complete. This database may be incomplete and the backup process cannot continue. You may delete the local database and attempt to repair it again.

The best way to recover from this (in my experience) is to just delete the offending backup version - no need to repair or recreate the database, and you only lose that one backup version.

In the Web UI, click on the backup set and then click “Commandline…”. Choose “Delete” from the dropdown Command list. Then scroll to the bottom and from the Add Advanced Option dropdown, choose “Version”. Type in “11” then click the Run button.

Edit: By the way, it is believed that this bug has been fixed in newer versions of Duplicati - but it hasn’t made it to the Beta channel yet.

Too late, I’m afraid.

I’ll try when this second attempt at recreating has failed (some hours more).

Some ways to avoid future problems of this sort are at Help with Unexpected Difference in Fileset however you may lack logs to check unless you have some other records besides what was in DB before Recreate.

You can decide whether a fairly well proven (for a canary) 2.0.4.22 or less proven 2.0.4.28 canary, or auto-compact turn off suit you… Basically, this is an old bug that’s fixed, but getting it in a true beta’s been slow. v2.0.4.23-2.0.4.23_beta_2019-07-14 is just v2.0.4.5-2.0.4.5_beta_2018-11-28 plus Amazon Drive warning.

I’m returning back to this after a few months being to busy with other priorities (I could live without this backup running as it is experimental/testing and my other machines/users don’t have the problems (they have other Duplicati hiccups though but sofar with successful repairs if need be).

Because I forgot where I was, I did a delete and rebuild again. That is still running (runs for many hours using 1-2 cores on my Core i7) and will probably fail again after many hours chugging along.

So, then I need to try the “Version 11” thing suggested above, Now, when that fails, is my backup completely hosed? All my history lost? Just throw everything away and forget about it? Because if that is the case, I’m starting to wonder if using Duplicati is really a good idea. I have some family members using it (some still on 2.0.3.3 beta because I stopped updating them to a newer version when I ran into this issue). A completely hosed backup is something that is so fundamentally wrong that I need to reconsider using Duplicati. I like the approach, I like the community, but I’m getting worried because of this issue and the other hiccups.

I’m a bit confused - is the database recreate still running? Yet you tried to delete a specific backup version? Or are you speaking in hypothetical terms here?

The database recreation process taking a long time may be due to a bug that has been fixed but not yet available in the beta channel. Same with the bug about ‘unexpected difference in fileset’ - not available in the beta channel yet.

If you are willing, I might suggest you run the latest canary 2.0.4.33 on this machine where you have a database recreation issue. See if it recreates the database more quickly. You could continue to use it until the next beta version is released, then switch back to that channel.

In my experience, the latest canary versions are better than the 2.0.4.x beta releases. I’m hoping a new beta will come out soon.

The database recreate fails with “The database was attempted repaired, but the repair did not complete. This database may be incomplete and the backup process cannot continue. You may delete the local database and attempt to repair it again.”. This time, maybe the fact that the computer went to sleep while repairing may have been stopping the recreate process, I don’t know and I’m loathe to try again because the whole process takes 30 hours or so.

Every time I try to solve my problem, it takes ages and it fails. I am confused as well and the backup hasn’t been running for months now, because (a) I don’t know what to do and (b) whatever is suggested does not work. The database has been deleted (again, that is the suggestion by Duplicati itself, if repair does not work, try delete and repair —I guess that means “delete and recreate from the backup”), so I guess it is gone.

Anyway, I might try the canary release. No going back from that as well, I know, hence it is another suggestion without a roll back. What I should have done in the first place is save the database so I could go back to an earlier version (maybe Duplicati should be very careful about actions that cannot be rolled back and always create a saved state for that, e.g. save a copy of the database before proceeding and rolling it back when some repair fails so that another attempt can be made from the original situation). Maybe.

At this stage, I’m starting to believe I have lost my entire backup, all the history in it etc. And I’m wondering how I will have to remove the actual index and data files from the S3-compatible storage. And what to use for off site backup by friends who back up to my S3-compatible storage. I can’t let them use something that is unreliable. But there are no really good solutions for macOS.

Duplicati does exactly that. Before each upgrade of the database, it makes a copy with a timestamp in the filename, and places it in the same folder as the database. This allows you to roll back without any hassle (unless you start making backups with the new database and then regret later).

I wasn’t talking about an upgrade, but about (delete &) repair

1 Like

This has been one of the main development issues too, along with too few volunteers in all areas. Probably there are also some process issues as well. Regardless, Beta code base is a year old… Numerous fixes to prevent backup breakages (and allow fast Recreate if need be) are not in Beta.

Eventually new Beta should come. 2.0.4.33 canary is probably very good, although also very new.
Major area of worry to me personally is that the Stop button change in 2.0.4.29 adds lots of issues.

For your experimental/testing machine, a more potentially surprising release like a canary might fit, however if you’re using it as a proxy for family backup (including fixes), sharing pain might be best.

Generally my advice on reliability is that Beta code is, pretty much by definition, still not fully stable, meaning it should not be the only backup for anything you’d seriously hate to lose. Restarting fresh sometimes is the best path for future backups, however there are numerous levels of recovery from problems if restores are what you want. Duplicati.CommandLine.RecoveryTool.exe uses a different restore method than the usual GUI or CommandLine restores, and is more tolerant of any problems. Beyond that, there’s a Python script mainly intended to have no dependency at all on Duplicati code:

Independent restore program (yet another thing to test with new Stop code – I tested RecoveryTool).

If you still have files on the destination, you might be able to recover your backup, but whether or not you can continue will depend on test. The recent Canary Recreate is usually vastly faster than before.
It can’t overcome all possible errors though, and only more test will find out how healthy destination is.

You can also get a preview of whether or not it will work by running Canary on some system to try the direct restore. If upgrading a system on Duplicati now, downgrading can be harder, due to DB format changes. This is where the backup files mentioned can come in handy. Upgrade via GUI then a quick downgrade by manual removal of the upgrade (and use of a backup DB if needed) if you like will work.

Downgrading / reverting to a lower version talks about this, but simpler is upgrade-and-stay-upgraded-until-Beta, or finding some other system to play with, where there’s nothing you care about in Duplicati.

I haven’t tested, but for Recreate button (as opposed to a DIY), the plan for allowing rollback may differ: