Local backup - Corrpution after a week - unreliable?

I’ve a small local backup on a disk which has not been disconnected (Win10)
It’s just 15GB raw size, 6GB compressed.

It was running for about one week, today I got corruption errors: a file is supposed to be missing and the recommended “repair” failed as well.
Here:
Jan 23, 2021 1:01 PM: put duplicati-b8929c031d1fe4c3d8ae89d7293edbc2a.dblock.zip.aes
{“Size”:43181,“Hash”:“N9WUAQjA3Y3a/xCiQMT44P5iL0dHByz5Lx2ykXVp3Aw=”}

This file was supposed to be on the disk but it’s not there.

Welcome to the forum @nolife

Truly local files are one of the most reliable destinations. If a file goes missing, the error should look like:

When starting backup, a GUI popup will say something like the below. It sounds like you saw this popup:

image

A <job> → Show log entry for that time is not created, but a Home → About → Show log one is made:

Jan 25, 2021 12:27 PM: Failed while executing "Backup" with id: 2
Duplicati.Library.Interface.UserInformationException: Found 1 files that are missing from the remote storage, please run repair
   at Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList(BackendManager backend, Options options, LocalDatabase database, IBackendWriter log, IEnumerable`1 protectedFiles)
   at Duplicati.Library.Main.Operation.BackupHandler.PreBackupVerify(BackendManager backend, String protectedfile)
   at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at CoCoL.ChannelExtensions.WaitForTaskOrThrow(Task task)
   at Duplicati.Library.Main.Controller.<>c__DisplayClass14_0.<Backup>b__0(BackupResults result)
   at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action`1 method)
   at Duplicati.Library.Main.Controller.Backup(String[] inputsources, IFilter filter)
   at Duplicati.Server.Runner.Run(IRunnerData data, Boolean fromQueue)

The two log situation is awkward, but you actually need a third log to see what the missing file name was.

About → Show log → Live → Warning watching the backup will identify what specific file was not there:

Jan 25, 2021 12:27 PM: Missing file: duplicati-b0b5374b518744eddaba2ac2ede336d02.dblock.zip

I’m not sure what you’re showing in “Here:”, but it looks like <job> → Show log → Remote showing file’s initial upload. If looking at that log, you also have to make sure it wasn’t deleted later, e.g. due to Compact. How did you come to pick the file you showed? Perhaps there’s a better way, or yours behaved differently.

But if you got the red popup, that should have taken such intentional deletions into account before an error.

Destination files are verified frequently for presence. How often do backups run, and were there any before error and after Jan 23, 2021 1:01 PM? That would narrow down when the file vanished. Any reboots done?

EDIT:

I should add that local files sometimes trigger antivirus activities, so that might one downside to local files.
Using encryption can help with that, because the files will look like random data, so AV won’t find anything.

Hi,

I identified the filename through the logs already, it’s not there indeed.
But nothing deleted it, no antivirus was active, the hdd was always on and ready.
Also it is AES, standard options
The computer wasn’t even rebooted.

In 20 years I’ve not had a file “vanish” for no reason from a brand new flawless disk.

Duplicati did either remove the file or it never created it, then it complained about it missing and failed to repair.

That’s a dead end showstopper, if Duplicati can’t stay stable in the most simple and reliable environment for ONE WEEK how could you ever rely on it restoring anything ?

Or it created it, deleted it in compact, and lost track of the deletion. I’m still wondering what logs you have.
Because files come and go, history matters. One not-huge log is log-file=<path> at log-file-log-level=retry.

Creating a bug report and posting a link to it is a way for the experts to get insight into history of given files which might have to be done here unless you think you can set up a log file and reproduce the issue for it.

EDIT:

Was Duplicati also up the whole time, without process or job stops? I’m looking for what might have set this off. It’s certainly not a typical thing, otherwise there would be a lot of reports, and I’m not seeing that.

Hi,
Duplicati was up, the PC was up, the disk was connected, no reboots.
This happened within 3 days, the logs didn’t show anything except that the file was created and next day during backup it was reported missing.

I fear that a lot more than a bug-fix is needed. Something like that should never happen to a backup storage. Not even when it runs tens of thousands of backup, let alone 3 backups.
I fear at this point, even if it’s a single case, the software is disqualified for backups.
Which is a shame given the ideas behind it

Making progress. I didn’t get the entire answer, but it sounds like no special actions happened.
Seeing “PC” makes me think “Windows” which makes me think of Defender. Is that disabled?
How do I check my backup is integral? - Antivirus deleted some archives is why I’m pushing…
That might be the most recent case of such a thing. I know I’ve had AV delete my files before.

I certainly don’t know every forum post, but this missing file seems a hugely unusual situation.
https://usage-reporter.duplicati.com/ shows statistics of 1.5 million backups per month to files.
I’m trying very hard to collect information here, because I don’t think there’s any general issue.
There are definitely issues, as you can see from this forum. That’s a reason it’s still in Beta…

That weakens the compact theory. Barring special options, most recent backup is not deleted.
Deleted backups are what trigger a compact. It might still be useful to check last good backup.
Clicking on Compact Phase will show if anything happened. Better DIY logs have more details.

There are definitely heavy duty ways to track down odd rare issues, but it takes some doing…
Rather than go right there, if you’re willing you can post the bug report though it might not help.

Agree with the “should”, but getting there takes work, including by people who hit such issues.
Until the software is perfect (never), a less-than-ideal solution is repair tools and restore tools.
The LIST-BROKEN-FILES command can show what effect missing file has, in source terms.
The PURGE-BROKEN-FILES command can’t bring back lost data, but can get backup going.
Any source files that were damaged will lose their old versions, but the backup will get current.
Disaster Recovery talks about getting up, and also about giving up, and recovering what’s left.

Having multiple backups is good practice. Some run Duplicati as a secondary. You could try it.
I’m not sure how much of a fluke this one was, but there’s not enough info yet to find cause…

Having written all the above (please read), I thought of a better test. <job> → Show log → Stored possibly is where your quoted log info is from (if not, where?). At the end of a backup is a list and typically three get operations which are the post-backup verification of sample-testing of a configuration number of files.

The list is recorded in the database, which is why a database bug report would help, but you can click it.

Here’s one of mine:

Jan 25, 2021 10:55 PM: list
[
{"Name":"duplicati-20210126T035430Z.dlist.zip","LastAccess":"2021-01-25T22:55:08.7045899-05:00","LastModification":"2021-01-25T22:55:08.7045899-05:00","Size":94630,"IsFolder":false},
{"Name":"duplicati-b0ee3059fa96e447f97d8010cf0e0434b.dblock.zip","LastAccess":"2021-01-25T22:54:55.8090194-05:00","LastModification":"2021-01-25T22:54:55.8090194-05:00","Size":52428934,"IsFolder":false},
{"Name":"duplicati-b63f23f894a2c46869ccbaf42a9f6691f.dblock.zip","LastAccess":"2021-01-25T22:54:55.821711-05:00","LastModification":"2021-01-25T22:54:55.821711-05:00","Size":52428957,"IsFolder":false},
{"Name":"duplicati-b687a012aeca64445bb79068bf2230125.dblock.zip","LastAccess":"2021-01-25T22:55:06.9308104-05:00","LastModification":"2021-01-25T22:55:06.9308104-05:00","Size":3347351,"IsFolder":false},
{"Name":"duplicati-i41fab86b29514d09bfa8e2016ea8bf5e.dindex.zip","LastAccess":"2021-01-25T22:55:01.3125339-05:00","LastModification":"2021-01-25T22:55:01.3125339-05:00","Size":3116128,"IsFolder":false},
{"Name":"duplicati-i6c6d35451bad4913a3ab96e1c61ce710.dindex.zip","LastAccess":"2021-01-25T22:55:08.5428042-05:00","LastModification":"2021-01-25T22:55:08.5428042-05:00","Size":200579,"IsFolder":false},
{"Name":"duplicati-ie0df737be56d42e8a9747f898e9f8dd5.dindex.zip","LastAccess":"2021-01-25T22:55:00.3176919-05:00","LastModification":"2021-01-25T22:55:00.3176919-05:00","Size":3109149,"IsFolder":false}
]

It would be helpful to know if Duplicati saw the missing file right after it wrote it. A DB bug report would be better, but might be more work and worry on your end. If Duplicati didn’t notice a missing file in its list it seems to partly defeat the point of doing the list, but at least it’s a scenario that might be possible to rig.

If it saw the file, then it was there, and maybe something about its characteristics will give clue to issue…