I all started with some hardware trouble on the ‘remote’ (same building for most) S3-backend side where the Duplicati backups of my users are hosted. I was able to recover data from that disk to another and restart the S3 compatible engine (MinIO), but one user had trouble with the backup. Duplicati complained about 1051 missing files and a repair did not have the desired effect.
Then I looked at the S3 backend and noticed that suddenly instead of one bucket for that user, there was a second empty one. This one had a name that consisted of the S3 access key of that user, followed by a dash followed by the original name. That second bucket was empty. But more weirdly, the Duplicati backup was using the empty one (e.g. it got exported when I exported the command line version of the backup). So, I thought, let’s tell Duplicati to use the original one. Which I did and apparently, the backup worked again. I wondered, had ‘repair’ created that other accesskey-bucketname bucket temporarily? Anyway, I thought, let’s remove that empty extra one. So, I used minio-mc and tried to remove that extra empty bucket. To my horror, MinIO mc removed both. I.e. I did something like mc rb accesskey-bucketname --force --dangerous and what MinIO did was remove both accesskey-bucketname and bucketname, that latter one containing all the remote Duplicati files.
renskelaptop:~ sysbh$ mc ls albus
[2021-12-27 22:36:31 CET] 0B **[snipaccesskey]-renskedocumentsanddesktop/**
[2021-12-27 22:29:31 CET] 0B **renskedocumentsanddesktop/**
renskelaptop:~ sysbh$ mc rb [snipaccesskey]-renskedocumentsanddesktop albus
***mc: <ERROR> This operation results in **site-wide** removal of buckets. If you are really sure, retry this command with ‘--force’ and ‘--dangerous’ flags.***
renskelaptop:~ sysbh$ mc rb [snipaccesskey]-renskedocumentsanddesktop albus --force --dangerous
*mc: <ERROR> Unable to validate target `[snipaccesskey]-renskedocumentsanddesktop`. Requested file `[snipaccesskey]-renskedocumentsanddesktop` not found*
**Removed `albus/[snipaccesskey]-renskedocumentsanddesktop` successfully.**
**Removed `albus/renskedocumentsanddesktop` successfully.**
Luckily, I still had the old disk from which I could restore bucketname so I did.
But, as since that time, one more successful backup had been executed, the Duplicati client database was now out of sync with the ‘remote’ S3 storage. So, I got an error again. It asked me to try repair. Which I did but that did not help (I’ve never experienced ‘repair’ to do anything every time I got into trouble, but that aside).
list-broken-files works and tells me 1329 client side files are affected.
purge-broken-files --dry-run works but with empty output.
purge-broken-files fails:
Listing remote folder ...
System.Exception: Unexpected number of remote volumes marked as deleted. Found 1 filesets, but 0 volumes
at Duplicati.Library.Main.Database.LocalDeleteDatabase+<DropFilesetsFromTable>d__5.MoveNext () [0x0027e] in <e60bc008dd1b454d861cfacbdd3760b9>:0
at System.Collections.Generic.LargeArrayBuilder`1[T].AddRange (System.Collections.Generic.IEnumerable`1[T] items) [0x0003d] in <ff4862557686417e948477704dd0dccf>:0
at System.Collections.Generic.EnumerableHelpers.ToArray[T] (System.Collections.Generic.IEnumerable`1[T] source) [0x00033] in <ff4862557686417e948477704dd0dccf>:0
at System.Linq.Enumerable.ToArray[TSource] (System.Collections.Generic.IEnumerable`1[T] source) [0x00018] in <ff4862557686417e948477704dd0dccf>:0
at Duplicati.Library.Main.Operation.PurgeFilesHandler.DoRun (Duplicati.Library.Main.Database.LocalPurgeDatabase db, Duplicati.Library.Utility.IFilter filter, System.Action`3[T1,T2,T3] filtercommand, System.Single pgoffset, System.Single pgspan) [0x0066f] in <e60bc008dd1b454d861cfacbdd3760b9>:0
at Duplicati.Library.Main.Operation.PurgeFilesHandler.Run (Duplicati.Library.Main.Database.LocalPurgeDatabase db, System.Single pgoffset, System.Single pgspan, System.Action`3[T1,T2,T3] filtercommand) [0x00000] in <e60bc008dd1b454d861cfacbdd3760b9>:0
at Duplicati.Library.Main.Operation.PurgeBrokenFilesHandler.Run (Duplicati.Library.Utility.IFilter filter) [0x0061a] in <e60bc008dd1b454d861cfacbdd3760b9>:0
at Duplicati.Library.Main.Controller+<>c__DisplayClass35_0.<PurgeBrokenFiles>b__0 (Duplicati.Library.Main.PurgeBrokenFilesResults result) [0x0001c] in <e60bc008dd1b454d861cfacbdd3760b9>:0
at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String[]& paths, Duplicati.Library.Utility.IFilter& filter, System.Action`1[T] method) [0x0026f] in <e60bc008dd1b454d861cfacbdd3760b9>:0
at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.Action`1[T] method) [0x00009] in <e60bc008dd1b454d861cfacbdd3760b9>:0
at Duplicati.Library.Main.Controller.PurgeBrokenFiles (Duplicati.Library.Utility.IFilter filter) [0x0001a] in <e60bc008dd1b454d861cfacbdd3760b9>:0
at Duplicati.CommandLine.Commands.PurgeBrokenFiles (System.IO.TextWriter outwriter, System.Action`1[T] setup, System.Collections.Generic.List`1[T] args, System.Collections.Generic.Dictionary`2[TKey,TValue] options, Duplicati.Library.Utility.IFilter filter) [0x00030] in <eb902dee3a2c4c3fbc8192580b4a42b2>:0
at (wrapper delegate-invoke) System.Func`6[System.IO.TextWriter,System.Action`1[Duplicati.Library.Main.Controller],System.Collections.Generic.List`1[System.String],System.Collections.Generic.Dictionary`2[System.String,System.String],Duplicati.Library.Utility.IFilter,System.Int32].invoke_TResult_T1_T2_T3_T4_T5(System.IO.TextWriter,System.Action`1<Duplicati.Library.Main.Controller>,System.Collections.Generic.List`1<string>,System.Collections.Generic.Dictionary`2<string, string>,Duplicati.Library.Utility.IFilter)
at Duplicati.CommandLine.Program.ParseCommandLine (System.IO.TextWriter outwriter, System.Action`1[T] setup, System.Boolean& verboseErrors, System.String[] args) [0x00342] in <eb902dee3a2c4c3fbc8192580b4a42b2>:0
at Duplicati.CommandLine.Program.RunCommandLine (System.IO.TextWriter outwriter, System.IO.TextWriter errwriter, System.Action`1[T] setup, System.String[] args) [0x00002] in <eb902dee3a2c4c3fbc8192580b4a42b2>:0
I looked at the Local Database page for this backup on this client, and the options for Reset, Save, Save and repair and Move existing database are grey and do nothing.
So, what should I do now? Trying to recreate would delete the current database without save option, so if it was needed for another restore later I might need it.
So, what I did was: low level copy of the sqlite database (unix cp). Then I tried to recreate it. That worked.
The question is this: should I have done things differently? Because it seems to me that having to reset to low level unix commands is not how I should have solved this.