Backup failing due to unexpected difference in fileset

Good thing you asked, and posted a picture. I corrected my previous post. There’s an extra step to sort the filesets so that the newest one winds up with the lowest record number. Our viewers both number from 1, so reading through the timestamp values (because I can’t sort your picture by clicking) says fileset 50 would be backup number 0 (i.e. the most recent one) in pretty much the entire rest of the UI except for the message…

Because this technique is kind of an experiment, you can also sanity-check it by putting the Timestamp into a converter from Linux time to normal clock time. In GMT, one converter states it’s October 4, 2018 4:00:00 AM, and you should see that as 0 at top of Restore dropdown, with whatever that time would be in your time zone.

0 is also attractive because it could be from new damage. I hope there’s not old damage behind this though. Although something falling apart in an old backup is probably possible, damage while creating a new backup seems more likely to me. When the dust settles (or maybe before), you might say if anything odd happened.

I’m not sure if anyone will want it, but if you use the job menu Reporting --> Create bug report you can save a sanitized version of your database to a local file, in case someone wants to see the details behind your error. Ultimately the wish would be to understand the issue well enough to reduce frequency and improve recovery.

Maybe the safest and most visible way to do the delete is in the job’s Commandline menu item after changing the command to delete, and adjusting the other fields to match the delete command syntax which (compared to backup) looks like you’d clear out the box with the sources files and paths. Also add --version=0, and to do this as carefully as possible, you can add --dry-run, check it, and give that a try first to see what would occur were it not for --dry-run. For good verbosity, add --console-log-level=Information. If it’s easier, you could add these to the job temporarily instead of doing manual entry at least twice (once for trial, and once for a repair).

I’m hoping this gets backups back in business without any further issues. If there’s anything absolutely critical from the current backup that you want to preserve, maybe restore (from a sick backup) is possible before… Best case would be if the backup is more for short-term needs, and we’re doing this in the hope of recovery without having to start completely over or try the painful database recreate. If you’re up to try this, please do.

Ok, I’ve initiated the ‘delete’ command for version 0. Sadly I didn’t add any extra logging, but I’ve gone into About → System Logs and viewed the live log. This is all I get at the moment:

  • 15 Oct 2018 21:12: Deleting 1 remote fileset(s) …
  • 15 Oct 2018 21:12: All backups to delete:
  • 15 Oct 2018 21:12: Backups outside of all time frames and thus getting deleted:

21:12 yesterday is probably about the time I kicked it off. Will see how long this takes to complete, but it’s been running for over 12 hours now…

Andy

Thanks for trying this out!

My guess is the delete went pretty quick and now you’re waiting for compacting to finish. If there anything new on the live log?

It didn’t go well. Duplicati seems to have crashed during the delete operation.

I’ve just restart it, and it’s now going through the pending backups that it missed. Will see what state it’s in after it’s finished.

I’m starting to wonder whether running a ‘Canary’ build is the right thing for me. How would I drop back to a ‘Beta’ release?

Andy

Downgrading / reverting to a lower version gives general procedures, however it can require manual work to downgrade databases, depending on how much change has happened since the old beta version. It’s much simpler if one is willing to export backup jobs then start over, but sometimes people value their old versions. Your case is different because you had a broken backup to start with, and possibly the crash made it worse. Beyond the documented server database downgrade, it looks like the job database format has changed too.

I’m going to look into downgrading procedures sometime, meanwhile you can consider the direction you like. Maybe you’ll have a better idea where things landed after all the action finishes. Concerning the right thing, canary is sometimes better (due to fixes) than beta, sometimes worse (new bugs), and always less “known”.

Hi,

Well, after the delete of version zero, my backup is now running successfully again.

Is there any way to check that all the files have been backed up successfully? I wonder if deleting the version would mean that some of the files aren’t backed up again?

Thanks

Andy

Deleting version 0 should just makes things look as if that version was never there. The checks for changes should be done against the database records for the new most-recent version, and might upload a bit more.

For checking backups, some may look at the Restore display to see if it looks as expected, some may rely on the default sampled testing done after a backup and settable (as high as they can stand) with –backup-test-samples, some may favor The TEST command periodically (the same as the automatic testing, but it’s run on demand instead of all the time), some may do sampled restores (recommended regardless of anything else).

Testing ALL files directly could take lots of space and download use, but I’ve done sanity checking on smaller backups just by restoring everything then checking Windows File Explorer folder information against original. Better tools exist if one prefers to compare folders more closely, e.g. to test that all file contents are identical.

I had the same error (version 2.0.4.2 Experimental).

Deleting whatever backup version didn’t help (version 0, the version it was complaining about, a range of versions around the version it complained about, any version in the SQLITE fileset table that contained the number in the error message anywhere).

A database repair din’t help either, the check files operation gave the same error.

At the end, delete and recreate database solved it for me.

If the delete testing used the version complained about, or an ID from the fileset tables, that may be incorrect assuming my analysis above is correct. It hasn’t been well-proven yet, but I’m not sure your deletes disprove. Right now, my fileset table has 49 entries with ID from 151 to 199. My theory is that the error messages would give the 151-199 number, whereas what the UI requires for input is 0 to 48 (zero-based version of 49 entries).

was hard to follow. You need to find the row based on ID, then compute a 0-based row number with 0 newest (which is opposite the way the database browser probably numbered rows, even ignoring the off-by-1 issue).

Did the fileset number in the complaint change? Did the number of entries in the Restore dropdown change? Seeing no changes might mean you have a backup like mine that’s run long enough to drop off old versions.

I don’t know the exact numbers anymore, but the record numbers matched the ID’s quite nicely, probably because no backup versions were deleted for this backup job.

Assuming that backup version 266 was the version it was complaining about:

  • I first deleted version 266. From that moment, the error messages reported a problem with version 265.
  • So I deleted version 260 to 270. These versions included all references to 265 (record number and backup version).
  • From this moment, errors were reported for (I guess) version 259.
  • I deleted the local database for this backup job an created a new one. DB creation completed successfully, I haven’t seen any error messages since then.

In summary, from what I remember, deleting any version from the backup set completed successfully, but resulted in an error message for one version newer than the one(s) I deleted. Repairing the database completed successfully, but didn’t change anything to the error messages. After a full recreate of the local DB, the backup worked without issues.

I found my workaround for the unsuitable-for-UI-use fileset numbers may be obsolete as of 2.0.3.12 canary:

https://github.com/duplicati/duplicati/compare/v2.0.3.11-2.0.3.11_canary_2018-09-05…v2.0.3.12-2.0.3.12_canary_2018-10-23

This fix is in 2.0.4.2 experimental too, and it might explain how the error message’s fileset number changes, for example if something newer than the bad one was deleted, the UI number of the bad fileset grows lower, however my claim remains that database ID numbers from the Fileset table can’t be put directly in --version because low numbers are the most recent in the UI numbering, but the oldest when looking at the ID field…

@kees-z was talking about complaint numbers, and also database records, so I’m not sure of exact steps, however if the message looked different than the old one (see source), and its number was used for delete but the fileset error remained (see first bullet), then I hope there’s not an issue with the new fileset mapper.

I know this is a fairly old thread but I experienced the same: difference in fileset. Tried a lot of solutions recommended here, but nothing worked. Then I looked in the folder containing the .sqlite databases and I found a database backup predating the fileset timestamp. I stopped the duplicati service, I moved the current backup database and renamed the backuped db to the name of the current db. Then I ran a backup. Everything smooth sailing from there.

Welcome to the forum @Gijs

f the backup DB is pretty recent, this might be a reasonable approach, though a future repair might delete some extra files that the stale DB didn’t know about. If the DB is very old, it might take a substantial amount of new backing up to get the DB and the backup files updated from whenever the DB was made. The main reason for the backup files is for a way to get back onto an older database format if a new Duplicati (e.g. a Beta) doesn’t run satisfactorily. Databases are upgraded automatically, but if you go back to older Duplicati, it won’t handle DB formats that didn’t exist when it was written. That’s the main use for the backup file – it’s in an older DB format.

Speaking of new Duplicati, at least one way to get this was fixed in v2.0.4.22-2.0.4.22_canary_2019-06-30:

Fixed data corruption caused by compacting, thanks @ts678 and @warwickmm

but the only way to get it (because 2.0.4.23 beta is basically 2.0.4.5 plus a warning) is a recent canary, e.g. v2.0.4.28-2.0.4.28_canary_2019-09-05 and then you can change Settings back to Beta channel, if you like. This error was happening to me way too much, so I was on 2.0.4.22 (which worked very well). 2.0.4.28 is not yet very well proven, but seems to be doing OK. At some future date it should lead to new feature beta.

1 Like

Hi @ts678, I’m already on the Canary build, due to problems with getting the beta (the one including the warning) to work with MacOS Catalinas stricter file system permissions. In fact, I think the backup was created approximately at the time of the upgrade.

I suspect that the database got corrupted when I was recording in Logic Pro X while the backup was running. Logic deleted files that were candidate for backup, something like that, but I’m not sure.

I don’t know if you can refine “approximately”, but FWIW the issue that got fixed is actually introduced by a compact in the backup before it’s detected, and that’s probably true in general, i.e. it’s a pre-backup check.

Looking through your job logs for runs which actually seemed to do a compact (i.e. it shows statistics) can isolate when the problem was introduced. Or maybe it’s some other cause, e.g. your thought on recording:

If you can figure out how to cause some error reliably, posting steps here or on an issue (they get tracked) would let someone try to reproduce it on their own system for debugging and hopefully a code fix for issue.

Hi,

Yesterday I got the same error.

Unexpected difference in fileset version 4: 13/11/2019 00:35:32 (database id: 96), found 139629 entries, but expected 139630

I didn’t try to delete the version yet as i don’t really know how to, and i don’t want to mess with my data.

Or maybe should i try to update to a beta or experimental version as i’m still on the default one (2.0.4.23_beta_2019-07-14), and then maybe come back to the default version?

Any idea about what i should do?

Thanks

It is not too difficult to delete the offending version. Here’s a quick list of instructions:

In the Duplicati Web interface, click your backup to expand all the options. Click “Commandline …” and change the dropdown to “delete”. Scroll to the bottom of the page and in the “Add advanced option” box, choose “version”. Type the number “4” in the version field to delete the version causing your issue and then click the “Run ‘delete’ command” button at the bottom of the page.

Upgrading to the latest canary won’t solve this problem in of itself, as far as I know. You still have to delete the offending version. But running the newer canary version should prevent this problem from happening again.

Note that if you do decide to go to the newest canary version, you cannot easily go back to the beta version you are running now. This is because the database schemas are changed. You would be able to switch back to the beta channel at the next beta release though.

1 Like

It worked! Thanks

Do i really need to install the canary? Is the move worth it? I’m scared that it will not be stable, i mean this message “Not for use with important data.” worries me.

Nope, you can stay on the version you are currently running. You may have this problem again, though, but in my experience I was able to resolve it each time by deleting the offending version.

Yep I understand. In general I would not recommend Canary releases for normal production use as it can have bleeding edge changes that are not well-tested. But the current 2.0.4.34 version is getting close to being promoted to Experimental/Beta channels, and works better than the latest beta release for most people.

There is only one known notable issue that is holding up the promotion to experimental/beta and it has to do with using the Stop command in the middle of a backup. Note that the latest beta also has a problem with that command.

Personally I never use that function so it didn’t worry me. I currently have 2.0.4.34 on all my production machines but I intend to go back to the beta channel on most of them once the next one is released.

Totally up to you of course!

makes me point to a similar concern for probably the same problem

See After update to latest beta, backups broken for my reply which is basically that canary varies, but you can keep an eye on them. Sometimes they’re on the verge of Beta, like now, sometimes far away. You can also test a canary in better or worse ways, and you can revert back to Beta channel and wait. Eventually a genuine fixed Beta should emerge (I sincerely hope – not seeing much progress though).

Beta for me was extremely unreliable due to “Unexpected difference in fileset” up until the 2.0.4.22 fix whose description points to a workaround if other old beta bugs don’t bother you and you have space:

–no-auto-compact added in Options → Advanced options then checkmarked will avoid compact bugs.