Can’t run backup — files are missing from the remote storage (google drive)

johnD · December 7, 2023, 8:06am

When starting a backup job I will get this error message:

When I click “repair” I will get this error message:

I don’t know where to enable ‘–rebuild-missing-dblock-files’ or how to “run the purge command”.

The remote target is google drive — this is my configuration:

{
    "CreatedByVersion": "2.0.7.1",
    "Schedule": {
        "ID": 2,
        "Tags": ["ID=6"],
        "Time": "2023-12-11T02:00:00Z",
        "Repeat": "1W",
        "LastRun": "2023-12-04T02:00:00Z",
        "Rule": "AllowedWeekDays=Monday",
        "AllowedDays": ["mon"]
    },
    "Backup": {
        "ID": "6",
        "Name": "…",
        "Description": "",
        "Tags": [],
        "TargetURL": "googledrive://…",
        "DBPath": "/config/CPGBQOIIIA.sqlite",
        "Sources": ["…", "…", "…", "…"],
        "Settings": [
            {
                "Filter": "",
                "Name": "encryption-module",
                "Value": "aes",
                "Argument": null
            },
            {
                "Filter": "",
                "Name": "compression-module",
                "Value": "zip",
                "Argument": null
            },
            {
                "Filter": "",
                "Name": "dblock-size",
                "Value": "50mb",
                "Argument": null
            },
            {
                "Filter": "",
                "Name": "retention-policy",
                "Value": "4W:1W,12M:1M,10Y:1Y",
                "Argument": null
            }
        ],
        "Filters": [],
        "Metadata": {
            "LastBackupDate": "20231127T020000Z",
            "BackupListCount": "10",
            "TotalQuotaSpace": "16106127360",
            "FreeQuotaSpace": "15841655165",
            "AssignedQuotaSpace": "-1",
            "TargetFilesSize": "203988178",
            "TargetFilesCount": "42",
            "TargetSizeString": "194,54 MB",
            "SourceFilesSize": "115899500",
            "SourceFilesCount": "4",
            "SourceSizeString": "110,53 MB",
            "LastBackupStarted": "20231127T020000Z",
            "LastBackupFinished": "20231127T020105Z",
            "LastBackupDuration": "00:01:05.8616670",
            "LastCompactDuration": "00:00:00.0085250",
            "LastCompactStarted": "20231127T020100Z",
            "LastCompactFinished": "20231127T020100Z",
            "LastErrorDate": "20231207T074914Z",
            "LastErrorMessage": "Found 5 files that are missing from the remote storage, please run repair"
        },
        "IsTemporary": false
    },
    "DisplayNames": {
        "…": "…",
        "…": "…",
        "…": "…",
        "…": "…"
    }
}

How to solve this issue?

gpatel-fr · December 7, 2023, 8:45am

this is done by using the CommandLine… (Kommandozeile …) option on the backup page. Click it, select ‘Purge-broken-files’, remove the source argument and all filters, click Run. Then do the same operation for repair.

johnD · December 7, 2023, 3:04pm

I have started the Purge-broken-files command hours ago and it is still running …

Does this make sense to you?

gpatel-fr · December 7, 2023, 6:03pm

Not really. Take a look at the live log (About - Uber - function/ Protokoll data anzeigen / live), select ‘Verbose’ level.

johnD · December 7, 2023, 7:21pm

So when I run this:

I am getting this

… and here the same but expanded:

What does it mean, and why does it suddenly happen?

gpatel-fr · December 7, 2023, 7:41pm

I don’t know but what you are doing is strange indeed. In your config you have set compression and encryption (the error messages talks about .zip.aes files) with a dblock size of 50 MB, and when running the purge-broken-files operation you are disabling compression and encryption and setting a dblock size of 100 MB. I don’t get why you are doing that.

ts678 · December 7, 2023, 10:19pm

I’m not sure where you see compression and encryption, but there’s a 100 MB in exampleOptionString, which comes up when the Advanced options is viewed as Edit as text and then completely wiped.

I don’t know why it was cleared. Perhaps it was a misunderstanding of the request to clear out its filters, which would probably just be --exclude or --include. Wiping out the whole box is going to be trouble.

Boolean options such as encryption are false by default, so noencryption is false thus encryption true. Attempting to run something so very invalid (options gone) in Windows doesn’t trip in libs, but does say:

ErrorID: EmptyPassphrase

Empty passphrases are not allowed

Return code: 100

How suddenly? Do you mean the purge-broken-files was the first time? Was backup doing well before? Backup day seems to be Monday, so I’m curious why there’s no Dec 4 backup shown in config posting.

What kind of system is this? It looks like one I’m not familiar with. What does mono --version say, e.g.

$ mono --version
Mono JIT compiler version 6.12.0.200 (tarball Tue Jul 11 21:33:00 UTC 2023)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
	TLS:           __thread
	SIGSEGV:       altstack
	Notifications: epoll
	Architecture:  amd64
	Disabled:      none
	Misc:          softdebug 
	Interpreter:   yes
	LLVM:          yes(610)
	Suspend:       hybrid
	GC:            sgen (concurrent by default)

Is Google Drive seemingly healthy with free space? There’s also your original problem that’s a mystery, which is why I’m again looking for some history on the system and any previous issues or repair efforts.

EDIT:

Maybe Dec 4 failed but wasn’t reported until Dec 7? Any system restarts or similar acts anywhere near? Upgrades and restarts can sometimes cause problems, but sometimes they can also remove problems.

gpatel-fr · December 7, 2023, 10:25pm

In the image in the first post, there is a reference to missing files named ‘.zip.aes’.

ts678 · December 7, 2023, 10:40pm

I should have quoted more.

I see how encryption was there, but not how it was disabled. Clearing all of the options leaves it on.
I think (not sure) that disabling compression would require a --zip-compression-level=0 option.

Maybe we can allow some more time for questions to be answered, and maybe a correct run done.
Recovering by purging files in the manual might help, although the default output is now a little less.

It’s probably not going to work with the library issues, so I’m interested in knowing more about them. There are other cases reported, although I’m not certain anybody has a really good handle on them.

gpatel-fr · December 7, 2023, 10:55pm

AFAIK it is not possible to switch encryption on and off on an existing backup.

ts678 · December 7, 2023, 11:10pm

My impression too, but I see no attempt to disable encryption. In my run, it’s still on and complaining

To add a little more value to post, I tested with Filters on the Source Data screen. As expected, got

ErrorID: FiltersNotAllowedOnPurgeBrokenFiles
Filters are not supported for this operation
Return code: 100

Either the include or exclude will cause above. I just have to click-to-remove them from the GUI view.
Many people won’t have any filters at all, but it’d be worth @johnD testing a correct purge-broken-files.
If it blows up in the libraries again, maybe restart Duplicati or the whole system to see if that helps them.

ts678 · December 7, 2023, 11:48pm

is the short story of a similar library issue, which is why I’m asking about the system and the mono on it.

The above was written in July 2022, so may have been 6.12.0.182, but current latest 6.12.0.200 claims

Fix a couple issues in the assembly loader

The complaints being seen are on what might be called libraries, but the proper term is really assembly.

Or it could be something else, but we didn’t write mono, so might as well take any possible fixes they do.

Right now we’re still waiting for a lot of answers about the system, Duplicati history, its mono, and so on.

johnD · December 12, 2023, 7:06am

Thanks for all your answers.

I am using duplicati 2.0.7.1_beta_2023-05-25
deployed via docker linuxserver/duplicati:2.0.7
and I am making backups to google-drive for a project since 26.4.2023 on every monday.
My last working, correct backup for this project was on 27.11.2023 and since 4.12.2023 I am getting these error messages (see above).
I have to mention that I am running 4 other backup jobs for other projects that just work fine.
But the one with the problems suddenly stopped working (4.12.) and I don’t know why. Every time I click Run now I am getting the same error response.

I have tried now to run the purge-broken-files command without changing anything (target url, advanced options, …), except wiping the commandline arguments. I had to do this twice and then it worked after I ran the repair command.

So, IT WORKS again.

But the question is still why did it happen?