Test command produces "Extra" errors

ts678 · April 27, 2022, 12:27pm

Not at all. If it were totally normal, it would normally happen, right? It’s rather hard to even force this.
Have you read through the issue?

Meaning what exactly? In the GUI, I think you get a red popup for an error, and yellow for a warning.
Some people (maybe including you IIRC) have opinions on levels. Some would like fewer warnings.

My own testing while researching to write the issue was a command line run. I don’t have old output.
I set up a fancier error by editing dindex file dblock index, fixing DB size and hash, and running test.

Running commandline entry
Finished!

            
  Listing remote folder ...
  Downloading file (647 bytes) ...
  Downloading file (647 bytes) ...
  Downloading file (694 bytes) ...
duplicati-i82e64e6790274843a13be40f297e66fc.dindex.zip: 2 errors
	Missing: 06CMxdyOi3kfKwq1k4XDtubnkvVDmN5/0AvCmK6+Tt8=
	Extra: 16CMxdyOi3kfKwq1k4XDtubnkvVDmN5/0AvCmK6+Tt8=

Return code: 3

I think the low return codes (maybe below 100) are not errors. I forget the exact return code scheme.

In the GUI, I ran Verify manually. The job has full-remote-verification on, and the Complete log said:

            {
  "MainOperation": "Test",
  "VerificationsActualLength": 3,
  "Verifications": [
    {
      "Key": "duplicati-20220427T113152Z.dlist.zip",
      "Value": []
    },
    {
      "Key": "duplicati-i82e64e6790274843a13be40f297e66fc.dindex.zip",
      "Value": [
        {
          "Key": "Missing",
          "Value": "06CMxdyOi3kfKwq1k4XDtubnkvVDmN5/0AvCmK6+Tt8="
        },
        {
          "Key": "Extra",
          "Value": "16CMxdyOi3kfKwq1k4XDtubnkvVDmN5/0AvCmK6+Tt8="
        }
      ]
    },
    {
      "Key": "duplicati-ba728e4517ca44f3193a344c400daabd8.dblock.zip",
      "Value": []
    }
  ],
  "ParsedResult": "Success",
  "Version": "2.0.6.102 (2.0.6.102_canary_2022-04-06)",
  "EndTime": "2022-04-27T11:40:57.5454052Z",
  "BeginTime": "2022-04-27T11:40:56.7553437Z",
  "Duration": "00:00:00.7900615",
  "MessagesActualLength": 9,
  "WarningsActualLength": 0,
  "ErrorsActualLength": 0,
  "Messages": [
    "2022-04-27 07:40:56 -04 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Test has started",
    "2022-04-27 07:40:56 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  ()",
    "2022-04-27 07:40:56 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed:  (3 bytes)",
    "2022-04-27 07:40:57 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20220427T113152Z.dlist.zip (647 bytes)",
    "2022-04-27 07:40:57 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-20220427T113152Z.dlist.zip (647 bytes)",
    "2022-04-27 07:40:57 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-i82e64e6790274843a13be40f297e66fc.dindex.zip (647 bytes)",
    "2022-04-27 07:40:57 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-i82e64e6790274843a13be40f297e66fc.dindex.zip (647 bytes)",
    "2022-04-27 07:40:57 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-ba728e4517ca44f3193a344c400daabd8.dblock.zip (694 bytes)",
    "2022-04-27 07:40:57 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-ba728e4517ca44f3193a344c400daabd8.dblock.zip (694 bytes)"
  ],
  "Warnings": [],
  "Errors": [],
  "BackendStatistics": {
    "RemoteCalls": 4,
    "BytesUploaded": 0,
    "BytesDownloaded": 1988,
    "FilesUploaded": 0,
    "FilesDownloaded": 3,
    "FilesDeleted": 0,
    "FoldersCreated": 0,
    "RetryAttempts": 0,
    "UnknownFileSize": 0,
    "UnknownFileCount": 0,
    "KnownFileCount": 3,
    "KnownFileSize": 1988,
    "LastBackupDate": "2022-04-27T07:31:52-04:00",
    "BackupListCount": 1,
    "TotalQuotaSpace": 999618043904,
    "FreeQuotaSpace": 129548185600,
    "AssignedQuotaSpace": -1,
    "ReportedQuotaError": false,
    "ReportedQuotaWarning": false,
    "MainOperation": "Test",
    "ParsedResult": "Success",
    "Version": "2.0.6.102 (2.0.6.102_canary_2022-04-06)",
    "EndTime": "0001-01-01T00:00:00",
    "BeginTime": "2022-04-27T11:40:56.7553437Z",
    "Duration": "00:00:00",
    "MessagesActualLength": 0,
    "WarningsActualLength": 0,
    "ErrorsActualLength": 0,
    "Messages": null,
    "Warnings": null,
    "Errors": null
  }
}

but there were no popups in the GUI, and the above not-user-friendly JSON looks happy with result.
Running a backup winds up the same way. These in my view are under-reporting the Missing block.

It’s not clear exactly what you would have liked if there was only an Extra, or if this really affects you.
The return code 3 affects my testing because it’s noise that I need to look at manually and not worry.

Likely a script could examine, but I didn’t write that yet. Detailed studies are more work, and I’m not positive Duplicati is doing a detailed study. It might just be reporting that it got some miscomparison.

Anyone who wants to dig into the code might start looking at the TestVolumeInternals code in file:
https://github.com/duplicati/duplicati/blob/master/Duplicati/Library/Main/Operation/TestHandler.cs

The possible enumerations (and some commentary about what they are meant for) might be below:

github.com

duplicati/duplicati/blob/beaf03562fdcf4425e962085bdf7175d6a465f49/Duplicati/Library/Interface/ResultInterfaces.cs#L269-L290


      
          /// <summary>

          /// The status of a change in a test entry 

          /// </summary>

          public enum TestEntryStatus

          {

              /// <summary>

              /// The element is missing

              /// </summary>

              Missing,

              /// <summary>

              /// The element was not expected

              /// </summary>

              Extra,

              /// <summary>

              /// The element was not the same as expected

              /// </summary>

              Modified,

              /// <summary>

              /// An error was encountered

              /// </summary>

This file has been truncated. show original

soko · January 7, 2026, 1:30pm

Hi @ts678

I just run into this “Extra” entries while running “test” via command line with full-remote-verification=true.

The github issue 4693 is closed since April 2025 so my version 2.2.0.1 from November should not produce such entries anymore.

The issue is quite long and very detailed (means: confusing ) so I hope its OK to ask here:

If full-remote-verification=true delivers only “Extra:” entry my backup is not corrupted and fully functional, correct?
How can I get rid of those extra entries? Is there a command who rebuilds the remote dblock files?

Thanks in advance.

Soko

PS: On my other backups I have outputs like the one below. Can I be sure that the “11 more” are also just “Extra:” errors?

duplicati-b783839425c1449b5952283dd5f5e3364.dblock.zip.aes: 21 errors
Extra: 1BmyKCYG3IBWhsWF/ZxCLUbqrKKdVfAnHkNGiykC5RE=
Extra: 1NIuvJqxVwzxJQzDJwLVq67i61wo0gDfIpJ6Oasj8sE=
Extra: 1ybNjXb8sYq8nWpYVkUOQA+iZmQNmf6t5oBH+TD9Dic=
Extra: 4IkIGdTwbjFqlZ/G0nyWfizmiDV60Xg0FYDR2FVKBc8=
Extra: 9MaC6K9vrowbELbCozhW2TpjnDULb/ejdKzWtCe1n/o=
Extra: CZuVCiuqz8aV6kxRiqLPsSwvX+gspWdqS0lMGOhb+7A=
Extra: F5pWKYIIJj4xYuoRcgjSqt1Jpr+yGlu33LB/YYXLWVE=
Extra: F9zu9nD5ZLZ6xuV3yVL4PBpQYqwDZU1C9cBzoF8Itcc=
Extra: GamxFFqQrw0FRwjzwCG2gHS/9YTmYFoFW2GgT4Vfu+I=
Extra: KrWiE7Dj8B80/n3DcqIt0u0NN/PHV+3pN+178qbbpCo=
… and 11 more

ts678 · January 7, 2026, 4:42pm

You can certainly ask, but you’re asking the person who simply filed that issue.
The person you need is the person who fixed it, but at least they left some info.

I think it says it should not produce such entries anymore on new dblock files.
Can you say, based on a look at the file dates, whether the files are old ones?
You’d probably also have to guess when you installed the version with the fix.
Dates on or near the program files can advise. There are harder ways as well.

This has long been the general idea. You should always actually test function.

What you probably need is not just a rebuild, but a rebuild with some filtering.

github.com/duplicati/duplicati

Improved number of blocks that can be removed during compacting

master ← feature/improve-compact-block-use-check

opened 03:48PM - 01 Nov 24 UTC

kenkendk

+186 -8

This PR implements the compacting fix from #4982 With this update the compact c…an exclude duplicated blocks by only checking if the current volume is providing them. The fix is not critical but can reduce the remote storage requirements in some cases. A test should be added before this PR is merged.

seems to suggest that if you force all files through compact, it will clean them.
That could be slow though, and can’t be specifically demanded. You can use:

  --threshold (Integer): The maximum wasted space in percent
    As files are changed, some data stored at the remote destination may
    not be required. This option controls how much wasted space the
    destination can contain before being reclaimed. This value is a
    percentage used on each volume and the total storage.
    * default value: 25

to make it extra sensitive (lower the 25) and click manual compact. I think raising
the remote volume size can also set this off, but then you’re left with big volumes.

soko · January 8, 2026, 6:18am

Thanks for the reply.

You basically approved my understanding on this issue. My “errors” are definitely from backups I’m running over 10 years now. So I guess they are from old files.

So re-compacting means the dblocks are downloaded, re-compacted and uploaded again I reckon. This is not feasible on my backups with some scratching on 1TB and my slow internet connection

ts678 · January 8, 2026, 6:20pm

You don’t need to guess. You can look, although it seems like a reasonable guess.

The COMPACT command. New UI click is “Compress now” not Compact. Uploads are not one-for-one with downloads, as the idea is to leave behind whatever data is no longer useful, as backups age, so it’s a repackaging, but still a lot of data move.

I’m not aware of anything that only “rebuilds the remote dblock files” with the Extra.

soko · January 9, 2026, 6:58am

I’ve executed the compact from the new UI on a small backup which has the Extra-entries in 3 dblock files, but it did do nothing:

Then I ran the compact-command from the command-line with the option --threshold=0 which downloaded, deleted and uploaded waaaay more then just the 3 dblock files with the Extra:-Entries.

Once it was done another test-run with full-verification had no “errors” anymore. So it seems to fix it.

ts678 · January 9, 2026, 1:40pm

Did you change threshold in the Options first? If not, it was a normal compact check, and that runs after every backup unless no-auto-compact option is set.

Compacting files at the backend

The compacting procedure is triggered after each backup, but can be disabled with an advanced option. If you want to perform a compacting operation manually, click on the backup name and click Compact now under Advanced.

As mentioned, the new UI calls it “Compress”, but I hope it respects set options.

It knows nothing about Extra count. It’s removing wasted space to the extreme (because it was asked to). It also filters away the duplicates while repackaging.

You might have been able to lower threshold gradually, but then you’d need repeated testing (or maybe you do test anyway) to see if Extra went to none.

Regardless, you went for it in one shot and it worked, so hopefully it stays good.

soko · January 9, 2026, 1:58pm

I was not able to change it. If you click:

It just starts. So I had to use the threshold advanced option via the command-line:

did I miss here something?

ts678 · January 9, 2026, 2:24pm

You change it in Options on the job first, like for pretty much anything except “Direct restore from backup files” (which has no job) and “Commandline” which starts from job options but almost always needs tweaking (for command to run, if nothing else).

was a bit too short I guess, but it meant job options:

soko · January 9, 2026, 2:37pm

I see!
So the advanced-options I set on the job itself are not just for the backup but also for the other commands? I did not know that

ts678 · January 9, 2026, 3:51pm

Yes. no-auto-compact is there. restore-legacy is there, and some options naturally apply to most commands. Examples: dbpath, passphrase, etc. are picked out of the full option set (rather than ask) when you do Compress now.

When you use Commandline GUI it defaults to backup and has most options, whether or not backup takes them all. Most commands ignore irrelevant ones.