"Completing previous backup" uploaded a synthetic filelist missing some metadata

Support
1

I was looking at “Completing previous backup” following update to latest (2.3.0.1). Did another kill-Duplicati-during-backup for a deeper look at synthetic filelist work.

What actually happened was an error during test:

2026-05-06 07:59:28 -04 - [Error-Duplicati.Library.Main.Operation.TestHandler-RemoteFileProcessingFailed]: Failed to process file duplicati-20260506T113128Z.dlist.zip.aes
SqliteException: SQLite Error 19: ‘NOT NULL constraint failed: Filelist-668F47926D3EB64C99900A7915CE035E.Metahash’.

  "TestResults": {
    "MainOperation": "Test",
    "VerificationsActualLength": 3,
    "Verifications": [
      {
        "Key": "duplicati-20260506T113128Z.dlist.zip.aes",
        "Value": [
          {
            "Key": "Error",
            "Value": "SQLite Error 19: 'NOT NULL constraint failed: Filelist-668F47926D3EB64C99900A7915CE035E.Metahash'."
          }
        ]
      },

An error also got flagged by the Python script I use to check backup after it runs:

    dlist_meta_block_set.add(file["metahash"])   # Doesn't support metablocklists.
                             ~~~~^^^^^^^^^^^^
KeyError: 'metahash'

and the reason for the error is that metahash and metasize aren’t in filelist.json:

"path":"C:\\Users\\Redacted\\My Drive\\Redacted's share\\desktop.ini","hash":"cYLfugaPWuC3+V1hHBIhNI2B2MhkKrlSUDTpwh7SlaA=","size":106,"time":"20260430T170333Z"},

I’m not seeing other reports of this, but for me it’s a 50% failure rate. It breaks DB recreate, but message says to list-broken-files and purge-broken-files.

Based on list-broken-files citing this file, I think purge will cure it, but I’ll wait briefly for any questions. I have a lot of logs and DB history and have looked a bit.

DB analysis started with

SELECT File.ID FROM FilesetEntry JOIN File WHERE FilesetEntry.FileID = File.ID AND FilesetEntry.FilesetID = 342 AND File.Path = "C:\Users\Redacted\My Drive\Redacted's share\desktop.ini"

which gave 21996 which I took to File to get MetadataID 19078 which I took to Metadataset to get BlocksetID 28407 which I took to BlocksetEntry to get BlockID 515644 which I took to Block to get a 149 byte block in VolumeID 7000 which I took to Remotevolume to find duplicati-b14695a13301f4aef95bfeeebe49784f4.dblock.zip.aes with State Verified.

Does that look right? If so, I wonder why the metadata didn’t get into the dlist?

Two interesting things about this file is it’s on a Google Drive sync, and timestamp somehow actually changed last night. I’m not sure how. Folder is otherwise empty.

EDIT 1:

No warnings or errors about metadata problems. Here’s the restart and what I had mentioned (plus usage reporter). This is from the server profiling log using regex:

^2026-05-06.*(ServerStarted|Warning|Error)

2026-05-06 07:34:10 -04 - [Information-Duplicati.Server.Program-ServerStarted]: Server has started and is listening on *, port 8301
2026-05-06 07:59:28 -04 - [Error-Duplicati.Library.Main.Operation.TestHandler-RemoteFileProcessingFailed]: Failed to process file duplicati-20260506T113128Z.dlist.zip.aes
2026-05-06 07:59:57 -04 - [Error-Duplicati.Library.Main.Operation.TestHandler-Test results]: Verified 3 remote files with 1 problem(s)
2026-05-06 08:03:10 -04 - [Error-Duplicati.Library.Modules.Builtin.RunScript-InvalidExitCode]: The script "HP4 clone 12 OneDrive\sync.bat" returned with exit code 4: Traceback (most recent call last):
2026-05-06 08:05:58 -04 - [Error-Duplicati.Library.UsageReporter.ReportSetUploader-UploadFailed]: UsageReporter failed

The recreate was actually done in a dummy job from a copy of destination files:

Error while running HP4 clone 12 OneDrive from sync

Recreated database has missing blocks and 1 broken filelists. Consider using “list-broken-files” and “purge-broken-files” to purge broken data from the remote store and the database.

As expected, there was the actual dlist from this run, in addition to the synthetic.
Hiding various combinations showed only the partial was bad, but oddly the DB looks like the current dlist has a different ID in File. It wouldn’t surprise me if the problem was also encouraged by whatever Google Drive is up to, but I’m having trouble figuring out how it managed to rather quietly correct that synthetic filelist.

EDIT 2:

To be clearer what I mean by “sync” is “mirror”, which “should” make files locally:

image
image465×362 23.8 KB

When I type the offending desktop.ini, it shows something maybe versioned:

[.ShellClassInfo] IconResource=C:\Program Files\Google\Drive File Stream\124.0.3.0\GoogleDriveFS.exe,27

That file creation date is April 30, but it started a new process on 5/5 6:25:28 PM.
It’s sort of close to the dir /as date. I looked for others near, but Everything
seemingly isn’t actually showing me everything. I’ve seen it get strange before…

05/05/2026 07:12 PM 106 desktop.ini

2

list-broken-files on the production backup (unlike test copy) sees no issue:

   Listing remote folder ... 
 Return code: 0

Database Repair button only has a warning about an old (2024) dindex problem.
So at least two of the first-test tools aren’t seeing the problem. Check dlist again.
Instead of rclone sync, get it from OneDrive in File Explorer. Change view editor
from notepad to Notepad++. The malformed entry still has the missing metadata.

Original error in TestHandler might have been from full-remote-verification
combined with its sample selection having picked up the dlist that got malformed.
I only have backup-test-samples of 1 though, as run-script-after tests too.

3

I did not manage to get a test to reliably reproduce this, but it looks like it happens when something is dangling in the database. When this happens, the SQL that generates the contents for the synthetic dlist is then generated, but it does not contain the metadata hash.

I can see that the code generating the dlist is explicitly handling this case, but I cannot figure out how/why we would ever record something without metadata. If the option --skip-metadata is set, all records are stored with the same “empty” metadata block.

And I assume it is there for all other entries?

So, you are saying that the database has metadata but the dlist does not include it?

Can you try on a setup where you have the error, to simply delete the dlist and then run repair, so Duplicati will re-create it? I am thinking that there is an order-of-operations issue where the problem is fixed, but the filelist is created before the error is corrected somehow.

I am pretty sure this issue occurs now because of the updated faster query. I have a potential fix for it, but I would like to understand what is causing it.

4

Seems so. Deleting bad filelist.json entry lets the checker script survive.

Yes, assuming the method that I posted is OK. Metadata matches repair result.

Repaired duplicati-20260506T113129Z.dlist.zip.aes has the missing metadata:

"hash":"cYLfugaPWuC3+V1hHBIhNI2B2MhkKrlSUDTpwh7SlaA=","size":106,"time":"20260430T170333Z","metahash":"S0EWCpHdRs+FoMk9jnyIWgMzGq5wxF53osQ+PBIZTP0=","metasize":149}

(matches what I found by hand as S0EWCpHdRs+FoMk9jnyIWgMzGq5wxF53osQ+PBIZTP0=)

Original duplicati-20260506T113128Z.dlist.zip.aes:

"hash":"cYLfugaPWuC3+V1hHBIhNI2B2MhkKrlSUDTpwh7SlaA=","size":106,"time":"20260430T170333Z"}

5

Thanks for the details. It looks like my idea was correct: the database gets repaired, but only AFTER the faulty list has been created.

I have made a small change that prevents Duplicati from ever writing an entry without a metahash, on the assumption that this should never happen. If it happens, we can at least trace the failure call and fix the issue.

If you can try your test again with the next canary, that would be great.

6

FWIW I’m not asking for a repair, and --auto-cleanup=false.
Is this some different sort of repair? What clue would log show?

In testing, at least two more tests with 2.3.0.1 ran without a fail.
2.3.0.102 got error first run, interestingly at same file as before.
One difference is the other one was near end. This was at end:

{"type":"File","path":"C:\\Users\\Redacted\\My Drive\\Redacted's share\\desktop.ini","hash":"IXsnjSc1blolgjzNAkDtlJyaRxCeNcU6mpmljI9VBrI=","size":106,"time":"20260513T182135Z"}]