X files found in remote storage, please run repair!

Support
1

Hi!

I run a 680GB incremental backup to a FTP remote site every two hours to backup my archive, and so far duplicati has worked very well in replacing Cobian.

Every once in a while though Duplicati will shoot this error, preventing me to leave this operation completely unattended.

How can I identify WHY this happens, and how can I automate the execution of repair? Thanks!

2

Hello @starkita and welcome to the forum!

The “X files not found in remote storage, please run repair!” message indicates that a bunch of files that Duplicati expected to find at the destination weren’t found.

Since Duplicati has to ask the destination for a list of files to compare to the local list, if the list isn’t returned or is only partially returned Duplicati will think files are missing from the destination.

If this happens with every run, then the files may actually be missing and you should click on one of the errors in your job log to get the specific file name then manually look on your destination for that file.

If it’s NOT there, then there’s not much that can be done - the file is gone.

However, if it IS there then the issue is most likely that Duplicati can’t see it for some reason. This might be due to a permissions problem (again, if this happens every run) or it could be a general internet glitch (if this only happens sometimes).

Also of note is if it’s the same files listed each time the error happens. If not, then it could be a destination problem (in the past there have been issues with file lists coming from some providers being incomplete or even having duplicate listings).

3

Hi!

This is the error:

Duplicati.Library.Interface.UserInformationException: Found 3 remote files that are not recorded in local storage, please run repair
in Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList(BackendManager backend, Options options, LocalDatabase database, IBackendWriter log, String protectedfile)
in Duplicati.Library.Main.Operation.BackupHandler.PreBackupVerify(BackendManager backend, String protectedfile)
in Duplicati.Library.Main.Operation.BackupHandler.Run(String sources, IFilter filter)
in Duplicati.Library.Main.Controller.<>c__DisplayClass16_0.b__0(BackupResults result)
in Duplicati.Library.Main.Controller.RunAction[T](T result, String& paths, IFilter& filter, Action`1 method)
in Duplicati.Library.Main.Controller.Backup(String inputsources, IFilter filter)
in Duplicati.Server.Runner.Run(IRunnerData data, Boolean fromQueue)

Once I run repair, everything is okay, but of course I would like to avoid errors

4

OK - so the issue is unexpected files FOUND in the remote location rather than ones NOT found there. (If you can confirm that then we should probably change the title of this topic.)

If you look in the job log you should be able to click on the Error entry for at least one of these files and get the name of the unexpected remote file.

If it’s a Duplicati related file (dblock, dlist, dindex) then there’s an issue with the backup process that needs to be addressed. If it’s not any of those, then there’s probably something else dropping files into that folder from time to time.

5

This is the first successful backup after running repair tonight (when it said it didn’t find 3k+ files!)

DeletedFiles: 0
DeletedFolders: 0
ModifiedFiles: 3
ExaminedFiles: 108599
OpenedFiles: 3
AddedFiles: 0
SizeOfModifiedFiles: 0
SizeOfAddedFiles: 0
SizeOfExaminedFiles: 735082488839
SizeOfOpenedFiles: 1658743
NotProcessedFiles: 0
AddedFolders: 0
TooLargeFiles: 0
FilesWithError: 0
ModifiedFolders: 0
ModifiedSymlinks: 0
AddedSymlinks: 0
DeletedSymlinks: 0
PartialBackup: False
Dryrun: False
MainOperation: Backup
CompactResults:
    DeletedFileCount: 0
    DownloadedFileCount: 0
    UploadedFileCount: 0
    DeletedFileSize: 0
    DownloadedFileSize: 0
    UploadedFileSize: 0
    Dryrun: False
    MainOperation: Compact
    ParsedResult: Success
    EndTime: 29/01/2018 01:06:36
    BeginTime: 29/01/2018 01:06:25
    Duration: 00:00:10.7259473
    BackendStatistics:
        RemoteCalls: 11
        BytesUploaded: 13051427
        BytesDownloaded: 65558770
        FilesUploaded: 3
        FilesDownloaded: 5
        FilesDeleted: 1
        FoldersCreated: 0
        RetryAttempts: 0
        UnknownFileSize: 0
        UnknownFileCount: 0
        KnownFileCount: 27668
        KnownFileSize: 737939284887
        LastBackupDate: 29/01/2018 02:00:00
        BackupListCount: 2
        TotalQuotaSpace: 0
        FreeQuotaSpace: 0
        AssignedQuotaSpace: -1
        ParsedResult: Success
DeleteResults:
    DeletedSets: [
        Item1: 2
        Item2: 27/01/2018 10:00:00
    ]
    Dryrun: False
    MainOperation: Delete
    ParsedResult: Success
    EndTime: 29/01/2018 01:06:36
    BeginTime: 29/01/2018 01:05:35
    Duration: 00:01:01.3349344
RepairResults: null
TestResults:
    MainOperation: Test
    Verifications: [
        Key: duplicati-i3703e37997af4b04a549b80c7028ba2b.dindex.zip
        Value: [
            Key: Extra
            Value: duplicati-b2451a13f0f9c45f98ea246c4a215b627.dblock.zip
        ],
        Key: duplicati-i3703e37997af4b04a549b80c7028ba2b.dindex.zip
        Value: [],
        Key: duplicati-i48dad3110d784036973e9a95f36a0b9e.dindex.zip
        Value: [
            Key: Extra
            Value: duplicati-b4940df55aeb1459fb99309fb37a6c1e3.dblock.zip
        ],
        Key: duplicati-i48dad3110d784036973e9a95f36a0b9e.dindex.zip
        Value: [],
        Key: duplicati-20180129T010000Z.dlist.zip
        Value: [],
...
    ]
    ParsedResult: Success
    EndTime: 29/01/2018 01:09:27
    BeginTime: 29/01/2018 01:07:44
    Duration: 00:01:42.7632904
ParsedResult: Success
EndTime: 29/01/2018 01:09:28
BeginTime: 29/01/2018 01:00:00
Duration: 00:09:28.6131010
Messages: [
    Deleting 1 remote fileset(s) ...,
    Deleted 1 remote fileset(s),
    Compacting not required,
    No hash or size recorded for duplicati-i3703e37997af4b04a549b80c7028ba2b.dindex.zip, performing full verification,
    No hash or size recorded for duplicati-i48dad3110d784036973e9a95f36a0b9e.dindex.zip, performing full verification
]
Warnings: []
Errors: []

Do you mean it is dropping files in the REMOTE folder?

I’m gonna check…

6

Sorry, that was a very poor choice of words on my part. I mean something else could be SAVING files into the destination folder so when Duplicati runs it sees files it’s not expecting.

It could also be that Duplicate saved those files there but then wasn’t able to record them in the local database. Knowing what the names of the files are (or where) would help identify which scenario is happening.

Of course if you’re getting working backups now then it sound like Duplicati resolved the issue.

When did the missing 3k+ files message show up?

7

Hi,

sorry for the month plus of absence.

I wanted to let you know that I moved the scheduling from once every hour, to once every 12 hours. That have seemed to fix it!

8

Glad you got it working, and thanks for letting us know!

I went ahead and flagged your post as the solution so if others run into this they can more easily know to try what worked for you.