OK - progress has been made! Just to be clear - I’m pretty sure I brought this on myself, and am slowly look just because I want to know how I managed to inflict it upon myself. 
To re-iterate, I started my backup with 7z (before I knew about the issues) and then switched to zip. So I now have:
- 88 dlist files (63 zip + 25 7z)
- 92 dindex files (92 zip and 0 7z)
- 92 dblock files (79 zip and 13 7z)
I suspect the all-zip dindex files are because at one point I tested deleting all my dindex files and doing a Database repair - and because I am (now) using zip, all the “fresh” dindex files came in as zip.
Now this MAY be the underlying cause of the issue - I’m not sure, but if I do a database Recreate with the above sets of files everything runs fine EXCEPT:
- I get a “Got 25 warnings” message. This happens to match the # of 7z dlist files I have PLUS the 5 shown in the log are all 7z:
2018-11-16 15:06:06 -06 - [Warning-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-FileProcessingFailed]: Failed to process file: duplicati-20170913T190637Z.dlist.7z.aes,
2018-11-16 15:06:06 -06 - [Warning-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-FileProcessingFailed]: Failed to process file: duplicati-20170921T180007Z.dlist.7z.aes,
2018-11-16 15:06:06 -06 - [Warning-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-FileProcessingFailed]: Failed to process file: duplicati-20171009T163147Z.dlist.7z.aes,
2018-11-16 15:06:07 -06 - [Warning-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-FileProcessingFailed]: Failed to process file: duplicati-20171016T180000Z.dlist.7z.aes,
2018-11-16 15:06:07 -06 - [Warning-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-FileProcessingFailed]: Failed to process file: duplicati-20171023T180000Z.dlist.7z.aes,
-
Running
Select * From "Remotevolume" Where "Type" = "Files" And "ID" Not In (Select "VolumeID" From "FileSet")on my database returns 25 rows (all 7z dlist files) -
Running
Select * From "FileSet"on my database returns 63 rows with IDs starting AFTER the highest dlist.7z ID (and of course 63 dlist.zip + 25 dlist.7z = my total of 88 dlist files)
So it appears that during the database Recreate, the RemoteVolume table DOES get the 7z files noted in it, but the FileSet table does NOT (likely part of the 25 “RecreateDatabaseHandler-FileProcessingFailed” warnings).
Because of this discrepancy between the FileList and RemoteVolume tables, the DropFilesetsFromTable method (that is called as part of retention policy cleanup) FAILS with "Unexpected number of remote volumes marked as deleted. Found 29 filesets, but 54 volumes" (which should probably be re-worded as "Unexpected number of remote volumes (dlist files) marked as deletable. Expected 29 but found 54").
Unfortunately, this WARNING causes the entire backup job to be flagged as unsuccessful so even though the backup worked (and is in fact restorable) it’s treated as a failed backup.
I don’t know if the dlist.7z files are ACTUALLY bad (meaning the Fileset table is correct) or there’s a bug in that code (meaning the RemoteVolumes table is correct) or something else completely - but at least it’s not a total black box anymore.
Hopefully I’ll get some time to to fine-tune the debugging in RecreateDatabaseHandler and do a local test to figure out exactly where the errors are coming from.
Of course I wouldn’t object if somebody else wanted to take a look at it as well… 