Backup Failure - Found # files that are missing from remote storage

Support
1

Hi,

I’m getting a failure, an example of which is below:

Failed: Found 885 files that are missing from the remote storage, please run repair
Details: Duplicati.Library.Interface.UserInformationException: Found 885 files that are missing from the remote storage, please run repair
at Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList(BackendManager backend, Options options, LocalDatabase database, IBackendWriter log, String protectedfile)

What I have tried so far -

Running the command line repair:
“C:\Program Files\Duplicati 2\Duplicati.CommandLine.exe” repair googledrive://Duplicati/myhost --passphrase=mypassphrase --authid=myauthid --dblock-size=50mb --exclude-files-attributes=Hidden,System,Temporary, --log-file=DuplicatiLog.txt --log-retention=1M

This did not work. So, I tried uninstalling Duplicati (choco uninstall duplicati), removing the path \Users\myuser\AppData\Local\Duplicati, and then removing all backups from googledrive://Duplicati/myhost. Re-running the backup did not work, giving me the same error. This is how I run my backup:

“\Program Files\Duplicati 2\Duplicati.CommandLine.exe” backup “googledrive://Duplicati/ic01” --authid=myauthid “C:\” --accept-any-ssl-certificate=true --backup-name=Cdrive --encryption-module=aes --compression-module=zip --dblock-size=50mb --keep-time=3M --passphrase=mypassphrase --disable-module=console-password-input --log-file=“C:\Program Files\Duplicati 2\DuplicatiLog.txt” --log-retention=1M --exclude-files-attributes=‘Hidden,System,Temporary’

The above works for about 2/3rds my Win10 Pro hosts. For the other third it is currently failing. I’m not worried about blowing away old backups and starting afresh. I suspect there is corrupted Duplicati cache of state on the client that is perpetually breaking these backups. Any ideas on what I should try next would be much appreciated.

Thanks,

Marshall

1 Like
2

Were the failing ones working before? If so, what changed? This acts like a backup that used to work suddenly lost view of its files somehow, but I want to make sure that those failing hosts once worked.
Trying to get something started that didn’t work before would possibly take a different diagnostic plan.

Old Duplicati versions used the roaming profile instead of local. You could look to see what’s in there.
If you can find a dbconfig.json file, look in it. That maps the destination URL to things like database.

You’ve maybe got a database that thinks it has files, but can’t see them (which I guess makes sense, because you deleted files off Google Drive, so I guess you’re trying to start the backups fresh again?).

3

Hi ts678,

The failing ones were working before a recent Windows update. B/C Windows updates are rather opaque to me, I’m not sure what changed on disk, but probably myriad things.

Like I mentioned I really care nothing about old backups, Duplicati DB state, or any other state saved to the client or in Google Drive. So, what I would like to know is the best practice for fully expunging all state from the host that could be causing the backup to fail, getting back to a tabula rasa. I have this suspicion that Duplicati is storing state somewhere that lasts over uninstalls and/or blowing away the database.

Searching C: drive for dbconfig.json did not turn up anything, nor did just the string ‘dbconfig’. I’ve only been running Duplicati in the last four months with version 2.0.5.1_beta_2020-01-18, so I’m guessing this is not old enough to have that file.

Thanks!

Marshall

4

Does anything come back after the run that imagines it has a DB? As you say, must be something around.

Sysinternals Process Monitor could be used to see what files Duplicati is accessing. I assume it fails fast.

Alternatively you could set it up to watch any programs accessing files that end in .sqlite, but others do.

You could look in C:\ProgramData\Duplicati and C:\Program Files\Duplicati 2\data for leftovers.

SYSTEM user profile can have info, but unless your user is SYSTEM, you probably can’t even look there…

EDIT:

You could also watch for accesses to dbconfig.json. I still don’t know why you’re not finding one of them.

This has always been CLI only, no GUI? dbconfig.json is for CLI. GUI is Duplicati-server.sqlite file.

5

Hi ts678,

On a host that is working, I do see the dbconfig.json file in the admin users home directory under \Users[USER]\AppData\Local\Duplicati. I’m using puppet to run choco to install Duplicati. Once it is installed, I then only ever interact with the host using the CLI tools to run a daily backup. Is there some part of this choco installing (or something) that doesn’t instantiate an initial default
dbconfig.json file?

Marshall

6

If you mean this, I don’t use it, or Chocolatey, and Duplicati team does not make it, so I have no details.

Are you sure there is one? It could get made as needed. But your problem appears to be a leftover one.
EDIT: and you might also have a leftover DB, especially if failure is fast and it couldn’t have done # files.

Are you saying you have absolutely no GUI access for debugging? If you have GUI, run Process Monitor.

EDIT 2:

If you have a phantom backup, I guess you could change your backup command to a find to see what versions it has (leave off the filename[s]) for starters). Any old backup dates? Any super-recent dates?

7

Hi,

I’m a Linux admin who’s a little slow in understanding procmon.exe. When you set up the filter to see the duplicati process, what is the name of the actual process that I’d want to monitor?

Thanks,

Marshall

8

It’s what you started (unlike Linux), so Duplicati.CommandLine.exe, but easy plan is to ask for anything containing duplicati (case-insensitive), and that should do the trick. There “shouldn’t” be anything else…

EDIT:

I guess I should warn you that there be a lot of file accesses that are just the process loading its code…