"Unexpected difference in fileset" when starting backup

Support
1

Recently, I have installed Duplicati v2.1.0.3 and have uploaded a total of around 2,5 TB to S3 Glacier.
As stated in the forum, I have set backup retention to „Keep all backups“, no-auto-compact to true and no-backend-verification to true.

During the upload oft he first 750 GB i get multiple warnings like
"2025-02-01 23:19:07 +01 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-DeleteFileFailed]: Failed to erase file duplicati-b84e44b21002149cbb73a16a1b38f527f.dblock.zip.aes, treating as deleted: User: arn:aws:iam::541299994389:user/Duplicati is not authorized to perform: s3:DeleteObject on resource: \"arn:aws:s3:::duplicati-htpc/duplicati-20250201T092719Z.dlist.zip.aes\" because no identity-based policy allows the s3:DeleteObject action\r\nAmazonS3Exception: User: arn:aws:iam::541299994389:user/Duplicati is not authorized to perform: s3:DeleteObject on resource: \"arn:aws:s3:::duplicati-htpc/duplicati-20250201T092719Z.dlist.zip.aes\" because no identity-based policy allows the s3:DeleteObject action",

as I have’nt set the policy s3:DeleteObject. After realizing this problem, I recently allowed the S3 user to delete objects.
During an upload on 4th of Februar I got this error, as the internet connection got lost.

 "Errors": [
    "2025-02-04 02:18:56 +01 - [Error-Duplicati.Library.Main.Operation.BackupHandler-FatalError]: Fatal error\r\nSocketException: Der angegebene Host ist unbekannt.",
    "2025-02-04 02:18:56 +01 - [Error-Duplicati.Library.Main.Controller-FailedOperation]: Die Operation Backup ist mit folgenden Fehler fehlgeschlagen: One or more errors occurred. (Der angegebene Host ist unbekannt. (Der angegebene Host ist unbekannt.) (One or more errors occurred. (Der angegebene Host ist unbekannt.)))\r\nAggregateException: One or more errors occurred. (Der angegebene Host ist unbekannt. (Der angegebene Host ist unbekannt.) (One or more errors occurred. (Der angegebene Host ist unbekannt.)))"
  ],

In the next days, I have started further backups with additional folders which completed without errors – just further warning, that files cannot be removed as oft he missing s3:DeleteObject action.
This afternoon I have finished uploading all data, when a further run without new folders or data results in this error:

"Errors": [
    "2025-02-11 16:32:41 +01 - [Error-Duplicati.Library.Main.Controller-FailedOperation]: Die Operation Backup ist mit folgenden Fehler fehlgeschlagen: Unexpected difference in fileset version 5: 03.02.2025 22:16:08 (database id: 5), found 92819 entries, but expected 114011\n\rUnexpected difference in fileset version 2: 04.02.2025 18:31:31 (database id: 6), found 92841 entries, but expected 114032\n\rUnexpected difference in fileset version 4: 03.02.2025 22:16:09 (database id: 7), found 92819 entries, but expected 114011\n\rUnexpected difference in fileset version 3: 03.02.2025 22:16:10 (database id: 8), found 92819 entries, but expected 114011\n\rUnexpected difference in fileset version 1: 05.02.2025 06:46:24 (database id: 9), found 94867 entries, but expected 116058\r\nUserInformationException: Unexpected difference in fileset version 5: 03.02.2025 22:16:08 (database id: 5), found 92819 entries, but expected 114011\n\rUnexpected difference in fileset version 2: 04.02.2025 18:31:31 (database id: 6), found 92841 entries, but expected 114032\n\rUnexpected difference in fileset version 4: 03.02.2025 22:16:09 (database id: 7), found 92819 entries, but expected 114011\n\rUnexpected difference in fileset version 3: 03.02.2025 22:16:10 (database id: 8), found 92819 entries, but expected 114011\n\rUnexpected difference in fileset version 1: 05.02.2025 06:46:24 (database id: 9), found 94867 entries, but expected 116058"
  ],

This error remains, even if i tried again. I already tried to update to the stable version, but this doesn’t change either.
Can anyone give me a suggestion, how to fix that?

Screenshot 2025-02-11 214153
Screenshot 2025-02-11 214153564×374 74.4 KB

2

Duplicati does not fully support immutable storage, and by extension does not fully work with Glacier at this time.

Something really strange is happening here. If I make the output a bit more readable, it is:

Unexpected difference in fileset version 5: 03.02.2025 22:16:08 (database id: 5), found 92819 entries, but expected 114011
Unexpected difference in fileset version 2: 04.02.2025 18:31:31 (database id: 6), found 92841 entries, but expected 114032
Unexpected difference in fileset version 4: 03.02.2025 22:16:09 (database id: 7), found 92819 entries, but expected 114011
Unexpected difference in fileset version 3: 03.02.2025 22:16:10 (database id: 8), found 92819 entries, but expected 114011
Unexpected difference in fileset version 1: 05.02.2025 06:46:24 (database id: 9), found 94867 entries, but expected 116058

First thing that is strange is that “fileset version 2” shows up with id 6, but it is created after version 3/4.

It is also strange that you have 3 filesets with 1 second in between. This indicates some retry that has re-uploaded a filelist.

Since the files cannot be deleted, perhaps there is some point where the left-over files were just treated as “good”. Not sure if this is related to the outage on the 4th.

I would suggest removing the versions that are reported faulty. There is no nice way to delete a specific version in the UI, so you have to use the commandline to do that.

There is a built-in commandline “emulator” in the UI you can use:

Screenshot 2025-02-06 at 22.08.57
Screenshot 2025-02-06 at 22.08.57982×792 46.9 KB

In the “Command” field, choose “delete”.
Remove the paths in the “Comandline arguments” field.

In the settings below, add the options:

--version=5,2,4,3,1
--dry-run=true

Make sure to check the checkbox after adding “dry-run”.

Then click “Run “delete” command now”.
This will perform the delete but not change anything.

If that looks correct (it should delete 5 files), record the names of those files somewhere.
Then repeat, but do not add “dry-run” to actually delete the versions.

Since you appear to have write-only storage, you now need the names of the files that were “deleted” but failed to delete. Go manually and delete these files from storage.

And you should now be back on track.