Unexpected difference in fileset, found N entries but expected M

Support
1

I’m getting this error message:

Unexpected difference in fileset version 5: 2019-07-30 12:00:00 (database id: 89), found 1584538 entries, but expected 1584544

I see there have been other reports of this, but rather inconclusive.

I’ve no reason to think the backup was interrupted. It’s all in the local network local via sftp. There is no log shown in the list for this time (06:00 ish this morning), either under General or Remote. but I get them mailed to me and that shows the below. It seems to indicate that the backup failed completely - the last success time shown is the previous one.

This is the second time this has happened, and the previous time it kept on happening (this time has only just happened, so I don’t know what will happen next). Previously I tried repairing the database, which had no effect. In the end I deleted the whole backup and started over, which has been fine for a few weeks now, until today. Having to recreate a backup is time consuming and catastrophic.

It’s due to run again in a few minutes. I’ll post what happens.

Failed: Unexpected difference in fileset version 5: 2019-07-30 12:00:00 (database id: 89), found 1584538 entries, but expected 1584544
Details: Duplicati.Library.Interface.UserInformationException: Unexpected difference in fileset version 5: 2019-07-30 12:00:00 (database id: 89), found 1584538 entries, but expected 1584544
   at Duplicati.Library.Main.Database.LocalDatabase.VerifyConsistency(Int64 blocksize, Int64 hashsize, Boolean verifyfilelists, IDbTransaction transaction)
   at Duplicati.Library.Main.Operation.Backup.BackupDatabase.<>c__DisplayClass32_0.<VerifyConsistencyAsync>b__0()
   at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass3_0.<RunOnMain>b__0()
   at Duplicati.Library.Main.Operation.Common.SingleRunner.<DoRunOnMain>d__2`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at CoCoL.ChannelExtensions.WaitForTaskOrThrow(Task task)
   at Duplicati.Library.Main.Controller.<>c__DisplayClass13_0.<Backup>b__0(BackupResults result)
   at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action`1 method)

Log data:
2019-08-01 06:02:38 +01 - [Error-Duplicati.Library.Main.Operation.BackupHandler-FatalError]: Fatal error
Duplicati.Library.Interface.UserInformationException: Unexpected difference in fileset version 5: 2019-07-30 12:00:00 (database id: 89), found 1584538 entries, but expected 1584544
   at Duplicati.Library.Main.Database.LocalDatabase.VerifyConsistency(Int64 blocksize, Int64 hashsize, Boolean verifyfilelists, IDbTransaction transaction)
   at Duplicati.Library.Main.Operation.Backup.BackupDatabase.<>c__DisplayClass32_0.<VerifyConsistencyAsync>b__0()
   at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass3_0.<RunOnMain>b__0()
   at Duplicati.Library.Main.Operation.Common.SingleRunner.<DoRunOnMain>d__2`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__19.MoveNext()
2

The 12:00 backup just did the same again - no log file in the list of log files, but emailed to me as before:

Failed: Unexpected difference in fileset version 5: 2019-07-30 12:00:00 (database id: 89), found 1584538 entries, but expected 1584544

The numbers are the same, which is curious as I’ve been working on the machine since then, but I may only have modified files, not created new ones.

So it looks like that backup is stuffed. I’m just trying a repair again to see what happens.

3

Repair results don’t suggest anything is amiss. Log below.

However, I then tried another backup and it also failed as before, just like last time this happened. Unless there is anything else to try, it looks like I have to recreate either the database or the whole backup. I’ll try the database first.

MainOperation: Repair
RecreateDatabaseResults: null
ParsedResult: Success
Version: 2.0.4.23 (2.0.4.23_beta_2019-07-14)
EndTime: 2019-08-01 12:31:35 (1564659095)
BeginTime: 2019-08-01 12:29:27 (1564658967)
Duration: 00:02:07.7677410
Messages: [
    2019-08-01 12:29:27 +01 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Repair has started,
    2019-08-01 12:30:20 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  (),
    2019-08-01 12:31:34 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed:  (12.90 KB),
    2019-08-01 12:31:35 +01 - [Information-Duplicati.Library.Main.Operation.RepairHandler-DatabaseIsSynchronized]: Destination and database are synchronized, not making any changes
]
Warnings: []
Errors: []
BackendStatistics:
    RemoteCalls: 1
    BytesUploaded: 0
    BytesDownloaded: 0
    FilesUploaded: 0
    FilesDownloaded: 0
    FilesDeleted: 0
    FoldersCreated: 0
    RetryAttempts: 0
    UnknownFileSize: 0
    UnknownFileCount: 0
    KnownFileCount: 13214
    KnownFileSize: 344901172358
    LastBackupDate: 2019-07-31 18:00:00 (1564592400)
    BackupListCount: 6
    TotalQuotaSpace: 0
    FreeQuotaSpace: 0
    AssignedQuotaSpace: -1
    ReportedQuotaError: False
    ReportedQuotaWarning: False
    ParsedResult: Success
    Version: 2.0.4.23 (2.0.4.23_beta_2019-07-14)
    Messages: [
        2019-08-01 12:29:27 +01 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Repair has started,
        2019-08-01 12:30:20 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  (),
        2019-08-01 12:31:34 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed:  (12.90 KB),
        2019-08-01 12:31:35 +01 - [Information-Duplicati.Library.Main.Operation.RepairHandler-DatabaseIsSynchronized]: Destination and database are synchronized, not making any changes
    ]
    Warnings: []
    Errors: []
4

Usually you can just delete the offending backup version. Search the forum - there have been other threads with more information.

Basic instructions - go to your backup job in Duplicati WebUI, click “Command Line…”, change the Command dropdown to “Delete”, go to the bottom of the screen and click the Add advanced option dropdown, select “Version” and then type in the number of the bad backup version. Then click the blue Run button.

Normally I only have to delete one version… good luck!

5

There have also been a number of fixes for this problem since 2.0,4.5 beta, but 2.0.4.23 beta is basically 2.0.4.5 plus an Amazon Cloud Drive discontinuation warning that needed to be made. You could also try:

v2.0.4.22-2.0.4.22_canary_2019-06-30

Fixed data corruption caused by compacting, thanks @ts678 and @warwickmm

which improved my backup from annoyingly common “Unexpected difference” to none at all in many runs. There was at least one earlier fix as well (having to do with mishandled network failure on uploading IIRC).

If you take the Canary plunge, you probably should change your update channel settings to Beta or maybe Experimental before some bleeding-edge Canary shows up that doesn’t run well. It happens sometimes…

EDIT: The newer release doesn’t fix the current damage which is checked before the backup, but it might reduce future cases. It probably won’t prevent all because this is a generic error that a self-check noticed.

6

Delete and repair database seems to have dealt with the problem(repair on its own didn’t). It took over 13 hours to run though.

7

When this happens to me a Delete operation is all I need… No Repair… Odd that you had to do both.

8

I didn’t try just delete. When repair didn’t help, I went straight for delete+repair.

I’m more concerned that it happens in the first place though. As you say it may already be fixed, but unless it keeps on happening I’ll wait for the next stable version release.

9

Next time do just Delete - don’t bother with Repair.

You are running version 2.0.4.23, which is almost identical to version 2.0.4.5. (It just added a warning for Amazon Cloud Drive users.) I believe there have been some advancements in other versions that help with the underlying problem. You might want to wait until the next Beta version, but in my experience 2.0.4.22 Canary is quite stable. (Even though it has an older version number than 2.0.4.23, it contains many more fixes…)

10

2.0.4.22 is basically the 2.0.4.21 experimental which might have gone beta except for some known bugs and a desire for some more test time. Note whatever channel you install from manually sets your update channel, so if you want to change back to beta channel in Settings, just do so after manual canary install.

https://github.com/duplicati/duplicati/releases

To see how far behind latest code 2.0.4.23 is:

v2.0.4.23-2.0.4.23_beta_2019-07-14
@kenkendk kenkendk released this 19 days ago · 716 commits to master since this release

v2.0.4.22-2.0.4.22_canary_2019-06-30
@kenkendk kenkendk released this on Jun 30 · 37 commits to master since this release

v2.0.4.21-2.0.4.21_experimental_2019-06-28
@kenkendk kenkendk released this on Jun 28 · 47 commits to master since this release

But ultimately it’s your choice what you want to run.