Unable to repair database

Been backing up with duplicati to S3 for over a year now.
Had a bit of an issue with duplicati today, was messing around and messed up my backups.
So I restored my db and settings from backups, I backup my database on a regular basis.

So now my database is out of date.

When I tried to run a backup it stated that I had additional remote files or something like that.

When I try and run a database repair I get the following No filelists found on the remote destination
If I go into the backup and click remote I can see the files.

I can not fire off any additional backups dues to this.

Any suggestions?

It says that with the recreate (delete and repair) choice also?

Looks like it only gets thrown for the following

if (!filelists.Any())
     throw new UserInformationException("No filelists found on the remote destination", "EmptyRemoteLocation");

There are possibly a few things that could cause that but I’m not sure what. According to the code above, things I’m throwing at it to cause it aren’t even doing so and thus are exiting differently.

It does look like it might need to be coupled with the local files to get the result you have. Remote only with correct local doesn’t appear to trip it. So possibly incorrect local plus correct remote. But, I’m not testing any further here right now.

You might be able to find more info in about → show log.

This is what is says in the log when I try and repair.

Duplicati.Library.Interface.UserInformationException: No filelists found on the remote destination
at Duplicati.Library.Main.Operation.RecreateDatabaseHandler.DoRun (Duplicati.Library.Main.Database.LocalDatabase dbparent, System.Boolean updating, Duplicati.Library.Utility.IFilter filter, Duplicati.Library.Main.Operation.RecreateDatabaseHandler+NumberedFilterFilelistDelegate filelistfilter, Duplicati.Library.Main.Operation.RecreateDatabaseHandler+BlockVolumePostProcessor blockprocessor) [0x0027b] in :0
at Duplicati.Library.Main.Operation.RecreateDatabaseHandler.Run (System.String path, Duplicati.Library.Utility.IFilter filter, Duplicati.Library.Main.Operation.RecreateDatabaseHandler+NumberedFilterFilelistDelegate filelistfilter, Duplicati.Library.Main.Operation.RecreateDatabaseHandler+BlockVolumePostProcessor blockprocessor) [0x00037] in :0
at Duplicati.Library.Main.Operation.RepairHandler.RunRepairLocal (Duplicati.Library.Utility.IFilter filter) [0x000ba] in :0
at Duplicati.Library.Main.Operation.RepairHandler.Run (Duplicati.Library.Utility.IFilter filter) [0x00158] in :0
at Duplicati.Library.Main.Controller+<>c__DisplayClass18_0.b__0 (Duplicati.Library.Main.RepairResults result) [0x0001c] in :0
at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String& paths, Duplicati.Library.Utility.IFilter& filter, System.Action1[T] method) [0x0026f] in <e4ba2224a87c42cbadcd524985619ee3>:0 at Duplicati.Library.Main.Controller.RunAction[T] (T result, Duplicati.Library.Utility.IFilter& filter, System.Action1[T] method) [0x00007] in :0
at Duplicati.Library.Main.Controller.Repair (Duplicati.Library.Utility.IFilter filter) [0x0001a] in :0
at Duplicati.Server.Runner.Run (Duplicati.Server.Runner+IRunnerData data, System.Boolean fromQueue) [0x003ad] in :0

Verify

Duplicati.Library.Interface.UserInformationException: Found 1774 remote files that are not recorded in local storage, please run repair
at Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList (Duplicati.Library.Main.BackendManager backend, Duplicati.Library.Main.Options options, Duplicati.Library.Main.Database.LocalDatabase database, Duplicati.Library.Main.IBackendWriter log, System.Collections.Generic.IEnumerable1[T] protectedFiles) [0x00103] in <e4ba2224a87c42cbadcd524985619ee3>:0 at Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList (Duplicati.Library.Main.BackendManager backend, Duplicati.Library.Main.Options options, Duplicati.Library.Main.Database.LocalDatabase database, Duplicati.Library.Main.IBackendWriter backendWriter, System.Boolean latestVolumesOnly, System.Data.IDbTransaction transaction) [0x00019] in <e4ba2224a87c42cbadcd524985619ee3>:0 at Duplicati.Library.Main.Operation.TestHandler.Run (System.Int64 samples) [0x000ba] in <e4ba2224a87c42cbadcd524985619ee3>:0 at Duplicati.Library.Main.Controller+<>c__DisplayClass30_0.<Test>b__0 (Duplicati.Library.Main.TestResults result) [0x0001c] in <e4ba2224a87c42cbadcd524985619ee3>:0 at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String[]& paths, Duplicati.Library.Utility.IFilter& filter, System.Action1[T] method) [0x0026f] in :0
at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.Action`1[T] method) [0x00009] in :0
at Duplicati.Library.Main.Controller.Test (System.Int64 samples) [0x0004b] in :0
at Duplicati.Server.Runner.Run (Duplicati.Server.Runner+IRunnerData data, System.Boolean fromQueue) [0x00423] in :0

I also tried a delete and repair.
And it gives me the same message. No filelists found on the remote destination

When I delete the database and recreate, pressing the two buttons manually.
I get the same message.

Are you telling me that when the local files don’t match the remote destination it will fail to repair?
Because the destination does have newer files more than likely.

If that is the case, isn’t that a little bit counter intuitive.
I think there should be a selection for source of truth, no?

Do you recall if settings ever used prefix option to have remote file names not start with duplicati-?
As @Xavron notes, the error seems to be saying you’re missing dlist files, but it might be naming.

Can you be more specific? The three main kinds of files have dblock, dindex or dlist in their names.
You’ll usually see lots of the first two (they’re from source files), and one dlist per version, with a date.

If you mean Duplicati GUI <backup> → Show log → Remote, that’s the old log from that stale database.
Are you talking about an old upload of something? You might also be able to see a recent list showing current files at the remote. Click on the list line. Do you have a convenient way to look at remote files?

Where this is heading is you might need to Recreate, not Repair. This will rebuild the database from the remote files, but the remote files have to be there, and the error message that you’re getting worries me.

Yes. This an expected mismatch coming from a stale database (local storage) seeing newer remote files. Possibly the message should ask for Recreate from current remote files, but that’s not current wording…

Without any info, I don’t know if the remote files got hurt, but looking over file listing will show what’s there.

Do you recall if settings ever used prefix option to have remote file names not start with duplicati- ?
As @Xavron notes, the error seems to be saying you’re missing dlist files, but it might be naming.

Looks like they are all dblock files in S3. Nothing else.
duplicati-b4c6a6ba89c2342f5b964ac2c865131a4.dblock.zip.aes

All 3 destinations have the same naming conventions. duplicati.*****.dblock.zip.aes
Did the dlist files somehow get deleted during delete and repair operations?

If you mean Duplicati GUI → Show log → Remote, that’s the old log from that stale database.

Yes this is what I mean.

Do you have a convenient way to look at remote files?

I do I can access the S3 bucket and see all the destination files.

Without any info, I don’t know if the remote files got hurt, but looking over file listing will show what’s there.

Running this in a docker container.
I got that warning about S3 and the underscores/dashes issue being deprecated.

Before reading the forum, I downgraded to lower version of canary than the db allowed.
Tried to run repair on that, nothing happened.
Then I restored sqllite and duplicati settings from a 3 day backup, backed up the current version.
On that previous settings backup only one backup worked because there was no activity on it and so I ran it. So now one the 3 day old restores has the current version on one of the 3 databases.
Then I read the forum and saw that just upgrading to the latest fixes the S3 warning message.
So I did that, and upgraded the container again.
Then I proceeded to restore the databases from the current version, which was just a copy and not an actual backup.
Now it’s all over the place and none of the backups work.
It keeps saying I need to run repair but nothing happens when I do.
:man_shrugging:
image

So at the end of the day I would be ok with recreating the files/backups in the destination because I want to move it from standard IA to Glacier. But this worries me a bit. Looking at the issues I have right now and missing files, how do I ensure this doesn’t happen again, more over how do I fix it if it does. I guess this is why I am here.

There should be a dindex for each dblock. The prefix is fine but it removes hope of mere bad names.

When you delete a version, the database and remote reflect that. For the remote, that’s a dlist delete.
Deleting all versions should be pretty hard to do ordinarily, because of the allow-full-removal safeguard:

--allow-full-removal = false
By default, the last fileset cannot be removed. This is a safeguard to make sure that all remote data is not deleted by a configuration mistake. Use this flag to disable that protection, such that all filesets can be deleted.

In the case of dlist files that Duplicati doesn’t “know” about (stale database), possibly those are deleted during Repair. tt generally deletes unknown files, and I’m not sure dlist files are exempted from deletion.

This seems unlikely because you didn’t go far back on the database, but I’ll ask…

What was your Backup retention setting before? If prior runs deleted files older than your restored DB, and Repair deleted files newer than DB, that might explain why there are no backup versions dlist files.

If that’s not it, then I don’t know why you have no dlist files. It’s common with an interrupted first backup, and there’s an experimental method to recover a bit from that and avoid reuploading all of the data blocks.
History is lost though, because the database and dlist files were the only places that knew old versions.

If none of the destinations have dlist files (example name duplicati-20210310T182943Z.dlist.zip), then Recreate won’t have a file list to Recreate. Duplicati can ordinarily recreate missing dlist files if the backup information it needs is in the database, but using an old database throws in a mismatch wrinkle…

If you really want to, you can look in the old database restore with sqlitebrowser to see what dlists it has. Creating a bug report and posting that for someone else to look at Remotevolume table would also work…

That also inteferes a bit with my reading of the chronology (thanks for providing), but it seems now instead of extra unknown files you somehow flipped over to the opposite situation of missing files. Those are fairly small numbers of missing files. You could watch About → Show log → Live to see the exact missing files. Maybe one will be a dlist, and looking at its filename will go with your recall of history to find error source.

I don’t use Docker, but I guess you know that keeping Duplicati data in a container complicates changing containers. Maybe that’s why you had to copy things back in after changing containers in your sequence?

What does “no activity” mean? No backup run, so DB still matches remote? No changes of source files?

Does restore from current version (which sounds sort of strange by itself) refer to “backed up the current version” earlier, before at least one backup (or maybe all three) was run? A DB revert seems like it should still be complaining about files it hasn’t seen. Somehow it looks like your DB got ahead of your destination. however you won’t know for sure until you see those file names, preferably dlists with recognizable dates.

Although it would be nice to figure out the exact sequence and what might have gone wrong at what point, presumably you want backups working again. I don’t know how much you care about your older versions, or the amount of data uploaded again to S3. Can you comment on your priorities, to try to steer next step?

It might also be possible to focus on getting going again, while preserving old data to try to look at later… Although it sounds like you don’t have log files, having old databases should reveal most remote activities. RemoteOperation table would be where you could likely see when and how lost dlist files were deleted.
Paging through the job’s Remote log works, but is harder. Example delete that my Backup retention did:

Mar 11, 2021 2:43 PM: delete duplicati-20210310T194001Z.dlist.zip.aes

Are you telling me that when the local files don’t match the remote destination it will fail to repair? Because the destination does have newer files more than likely.

I don’t have a test for that precisely yet… but if the remote files are in good shape, you can delete the local db files you attempted to use and hit repair. Duplicati definitely doesn’t need those and will create them as long as the backup solution appears in Duplicati. That part you might need but I didn’t test that yet either.

Looks like you need to delete the db file just before running repair (as in already be in Duplicati and ready to click repair) as it automatically may attempt something and that automatic attempt may be bad.

Also when doing this, it looks like maybe the repair might not work 100% perfect but should work quite well. I would have to debug the code here to see why it looks like certain files are being re-added.

This should get you back on track. So it sounds like your main problem was the db part of this:So I restored my db and settings from backups

If that still doesn’t work then I don’t know what you can do because it should work.

is a key question, especially on hearing

so I asked about dindex, but now ask about dlist too. Without all three, remote files are in bad shape.

Wasn’t this tried? One result that sounds wrong is after DB delete, Recreate button should be unusable.
You should only be allowed to Repair if you have no DB. Only active buttons will be blue and responsive.

I think we need to determine current situation more precisely, e.g. if there’s still any doubt about remote.

Sorry for the delay in reply. Been a bit busy lately.

It looks like there are no dlist or dindex files in any of the folders locally or remotely.

My retention was one copy, so I think the repair or delete and repair like ts678 mentioned must have purged the dlist and dindex files. I think it’s time to redo all my backups put them in glacier and keep at least 2 copies.

This was a good exercise. Now I know what to look for on the destination.

Thanks for all the help but without those files the backup is helpless as such I’m going to redo.
Luckily it wasn’t a disaster recovery situation.