Release: 2.0.9.107 (Canary) 2024-09-11

ts678 · September 21, 2024, 1:20pm

Looks like one update of an SQL string from double-quoted (now illegal) to single got missed.

"Remotevolume"."State" = "Temporary"

2024-09-21 09:09:28 -04 - [Profiling-Timer.Begin-Duplicati.Library.Main.Database.ExtensionMethods-ExecuteNonQuery]: Starting - ExecuteNonQuery: CREATE TEMPORARY TABLE "SwapBlocks-A593DDE69D1BCB4889F19349B5679F41" AS SELECT "A"."ID" AS "BlockID", "A"."VolumeID" AS "SourceVolumeID", "A"."State" AS "SourceVolumeState", "B"."VolumeID" AS "TargetVolumeID", "B"."State" AS "TargetVolumeState" FROM (SELECT "Block"."ID", "Block"."VolumeID", "Remotevolume"."State" FROM "Block", "Remotevolume" WHERE "Block"."VolumeID" = "Remotevolume"."ID" and "Remotevolume"."State" = "Temporary") A, (SELECT "DuplicateBlock"."BlockID", "DuplicateBlock"."VolumeID", "Remotevolume"."State" FROM "DuplicateBlock", "Remotevolume" WHERE "DuplicateBlock"."VolumeID" = "Remotevolume"."ID" and "Remotevolume"."State" = 'Verified') B WHERE "A"."ID" = "B"."BlockID";

github.com

duplicati/duplicati/blob/2827be6bb81a5694a9d6750a4b737fbb1f486253/Duplicati/Library/Main/Database/LocalRecreateDatabase.cs#L685-L697


      
                  public void CleanupMissingVolumes()

                  {

                      var tablename = "SwapBlocks-" + Library.Utility.Utility.ByteArrayAsHexString(Guid.NewGuid().ToByteArray());

          

                      // TODO: either hardcode all string constants or none

          

                      // The first part of this query swaps out blocks for non-present remote files with

                      // existing ones (as recorded in the DuplicateBlock table)

                      // The second part removes references to the non-present remote files,

                      // and marks the index files that pointed to them, such that they will be removed later on

                      var sql = $@"

          CREATE TEMPORARY TABLE ""{tablename}"" AS

          SELECT ""A"".""ID"" AS ""BlockID"", ""A"".""VolumeID"" AS ""SourceVolumeID"", ""A"".""State"" AS ""SourceVolumeState"", ""B"".""VolumeID"" AS ""TargetVolumeID"", ""B"".""State"" AS ""TargetVolumeState"" FROM (SELECT ""Block"".""ID"", ""Block"".""VolumeID"", ""Remotevolume"".""State"" FROM ""Block"", ""Remotevolume"" WHERE ""Block"".""VolumeID"" = ""Remotevolume"".""ID"" and ""Remotevolume"".""State"" = ""{RemoteVolumeState.Temporary}"") A, (SELECT ""DuplicateBlock"".""BlockID"",  ""DuplicateBlock"".""VolumeID"", ""Remotevolume"".""State"" FROM ""DuplicateBlock"", ""Remotevolume"" WHERE ""DuplicateBlock"".""VolumeID"" = ""Remotevolume"".""ID"" and ""Remotevolume"".""State"" = '{RemoteVolumeState.Verified}') B WHERE ""A"".""ID"" = ""B"".""BlockID"";

While waiting for developer input, how old are the dindex files whose dblock files are missing?

EDIT:

BTW a repro for the bug was to copy a dindex file into another backup, then recreate other DB.

Example usage of CleanupMissingVolumes:

github.com

duplicati/duplicati/blob/2827be6bb81a5694a9d6750a4b737fbb1f486253/Duplicati/Library/Main/Operation/RecreateDatabaseHandler.cs#L469-L471


      
          // In some cases we have a stale reference from an index file to a deleted block file

          if (!m_options.UnittestMode)

              restoredb.CleanupMissingVolumes();

You wouldn’t happen to be using log-file option, would you? Database logs are less reliable.