Unexpected difference in fileset version 0

Support
1

Hello, I am trying to use Duplicati to sync my over 400GB external hard drive to OneDrive in case of drive failure. I’m new to using this program, and I thought the entire drive was backed up over the course of two days, but the backup still says “Last successful backup: Never”, and on trying to back up again I got an error message:

Error while running h
Unexpected difference in fileset version 0: 9/3/2020 5:23:25 PM (database id: 2), found 418088 entries, but expected 418166

I have tried using the “Repair” option under Advanced > Database… > Maintenance, but it did not fix the problem. What should I do in this situation? I’d rather not upload the entire 400GB again, because that took over 24 hours.

I am using the latest beta (2.0.5.1_beta_2020-01-18).

2

Did it pop up any error or warning boxes on the web UI?

Did it leave a log with its info in <backup> → Show log?

About → Show log is an alternative spot to log an error.
Clicking on an error entry will typically expand to details.

Any idea from OneDrive’s web UI how much uploaded?
Mine gives a Size column which is an easy value to get.

Going into the folder and sorting by date, there should be a file with dlist in its name near the last upload.

3

The error I pasted in my previous post was from the web UI after a backup attempt.
In the logs, I can see 1 “Operation: Backup”:

Examined 344847 (415.64 GB)
Opened 125894 (160.42 GB)
Added 125894 (160.42 GB)
Modified 0 (0 bytes)
Deleted 0

The operation claims to have lasted 16 hours and backed up 160GB of files, but the backup was longer than that and in OneDrive I can see that there are 310GB of files. I also see a .dlist.zip.aes file in my OneDrive.
In that list of logs, there are also two "Operation: Repair"s which claim to be successful, with no errors or warnings.

Examined 0 (0 bytes)
Opened 0 (0 bytes)
Added 0 (0 bytes)
Modified 0 (0 bytes)
Deleted 0

In About → Show Log there are several “Failed while executing” errors. Most of these errors involve “Unexpected difference in fileset version 0”, but the very first one says “Could not find file 'C:\Users\Jacob K\AppData\Local\Temp\dup-bd077e18-6bcf-462c-9875-d70aedecd012” instead. The directory to backup is H:, not anything in C, so that seems odd.

Should I upload my full logs? If so, is there an easy way to get the logs as a bunch of files so that I can upload them instead of pasting several walls of text? Thanks.

4

Was this the one-and-only backup ever done? If not, some time and files are probably from another run.
"BackendStatistics" in Complete log will show you things like file uploads and deletions for the run.

“Could not find file" naming a temporary folder file is an unsolved mystery at backup’s last data uploads, possibly having something to do with an early error. Can you see “SpillCollectorProcess” in stack trace? That’s what picks up partially filled temp files left over from when the earlier parallel file-fillers were done.

was a similar problem on OneDrive, and I suspect timeout needs to get longer. Possibly yours does too, however so far the evidence hasn’t been posted. You might also spot “RetryAttempts” in the backup log.

You can drag-and-drop files into the forum if you like. If it doesn’t like the file type or size, it’ll let you know.
Or you can paste links to files. Or using three backticks (```) above and below creates a scrolling viewer.

I’m not sure logs will answer all the questions, but they might help. Creating a bug report is another path.
The “Unexpected difference” error is from an internal consistency check on the database before backup.

5

I’m pretty sure this was my only backup, although there’s a slight possibility my computer rebooted during the process, since it ran overnight. From “BackendStatistics”:

      "RemoteCalls": 3415,
      "BytesUploaded": 87281938315,
      "BytesDownloaded": 86285943,
      "FilesUploaded": 3335,
      "FilesDownloaded": 3,
      "FilesDeleted": 1,
      "FoldersCreated": 0,
      "RetryAttempts": 73,

Here’s the complete log (copypasted into a text file): completelog.zip (2.1 KB)

How would I look at the “stack trace”?

RetryAttempts is 73, not sure if that’s helpful though since it could matter how many retry attempts per file rather than through an entire backup (i.e. 73 for 1 file is a lot, but 73 for a large backup containing many files isn’t so bad.

Thanks for the help so far, and I’ll consider making a bug report if I’m not able to fix the problem.

6

“BackupListCount”: 2,

says that, as of this run, you have two backups. The 73 RetryAttempts are concerning because you might have an upload problem, but it might be as simple as a timeout (see linked topic) needing setting of longer http-operation-timeout than the default 1:40. You uploaded 87281938315 bytes in about 16 hours (possibly having gaps, so math may be off), which means about 12 Mbits/second. Any idea what speed you expect?

Remote volume size looks like you left it at the default 50 MByte, which “should” finish well under 1:40, but maybe sometimes it doesn’t. Regardless, you might need to set up some logging to see what retries were.

About → Show log → Live → Retry can see them (click error for details), or if that’s too tiring (intermittent issue), then log-file=<path> and log-file-log-level=retry will show them, with the exact reason detailed there.

Viewing the Duplicati Server Logs describes the live log, and also the Stored log where your error might be. Viewing the log files of a backup job often skips backup results log when backup fails, so check both spots. Possible you’ll find your “Could not find file” error there. If so, maybe you’ll find SpillCollectorProcess below. Setting it up again to watch with live log or log file is too much to ask at this time, but maybe log now exists.

Sorry to be so nosy about your Internet connection, but if it’s asymmetrical (many are faster on download), then that gives us more leeway to try things such as a database Recreate from the destination files, which typically is not a whole download but in bad cases it can be, depending on the condition of the backup files.

You’ve definitely got a database problem (internal consistency check failed), seem to have flaky uploading, and I’m not sure of a great way to tell the condition of the backup unless you want to approximate it using a Direct restore from backup files (which you can do from the same Duplicati by retyping some information), which might show whether you have a backup with timestamp of your last backup start. I suppose you can also just look at OneDrive, especially if you sort by date. Near its end should be a dlist file with UTC date.