TEST command don't work with "--version"

Support
1

Version: - 2.0.6.3_beta_2021-06-17

Without “–version=0”

Duplicati.CommandLine.exe test “file://E:\PC_Backup” all --dbpath=“c:\Duplicati_DB\ORDJTXXWKM.sqlite” --no-encryption=true

output:

  Listing remote folder ...
  Downloading file (7.33 KB) ...
  Downloading file (7.24 KB) ...
  ...
  Downloading file (49.96 MB) ...
  Examined 93 files and found no errors

CLI with “–version=0”

Duplicati.CommandLine.exe test “file://E:\PC_Backup” all --dbpath=“c:\Duplicati_DB\ORDJTXXWKM.sqlite” --no-encryption=true --version=0

output:

  Listing remote folder ...
No files examined, is the remote destination is empty?
2

I have never tried this, but from the definition, using ‘all’ imply that everything is tested, and --version=0 imply that only a part will be tested. Using both together is not defined in the documentation, but it can be interpreted in 2 ways:

  • all of version 0 will be tested
  • Duplicati will pick a random part of ‘all’ and then check only the fraction of this part belonging to version 0.

In the second way, it could be very likely that nothing will satisfy the criteria…

3

Welcome to the forum @datahunter

Backup Test block selection logic covers a few things, but unfortunately doesn’t get into version option.
I suspect there’s an actual bug here that nobody had noticed. If I use sample count of all, 1 or default:

Starting - ExecuteReader: SELECT “A”.“VolumeID”, “A”.“Name”, “A”.“Size”, “A”.“Hash”, “A”.“VerificationCount” FROM (SELECT “ID” AS “VolumeID”, “Name”, “Size”, “Hash”, “VerificationCount” FROM “Remotevolume” WHERE “State” IN (“Uploaded”, “Verified”)) A, “Fileset” WHERE “ID” IN (System.Object) AND “A”.“VolumeID” = “Fileset”.“VolumeID” ORDER BY “Fileset”.“Timestamp”

is what a profiling log shows, and the IN (System.Object[]) seems unlikely to be the correct query.
As a guess, it was probably trying to generate a comma-separated list of numbers, but didn’t manage.

might be where the seemingly bad SQL chunk is coming from, but I need a C# developer to explain why.
Replace string concatenation with StringBuilder. was a 2.0.3.7 change, but 2.0.3.6 doesn’t work either…

EDIT:

2.0.3.6 says (longer version here):

Starting - ExecuteReader: SELECT “A”.“VolumeID”, “A”.“Name”, “A”.“Size”, “A”.“Hash”, “A”.“VerificationCount” FROM (SELECT “ID” AS “VolumeID”, “Name”, “Size”, “Hash”, “VerificationCount” FROM “Remotevolume” WHERE “State” IN (?, ?)) A, “Fileset” WHERE “ID” IN (?) AND “A”.“VolumeID” = “Fileset”.“VolumeID” ORDER BY “Fileset”.“Timestamp”
ExecuteReader: SELECT “A”.“VolumeID”, “A”.“Name”, “A”.“Size”, “A”.“Hash”, “A”.“VerificationCount” FROM (SELECT “ID” AS “VolumeID”, “Name”, “Size”, “Hash”, “VerificationCount” FROM “Remotevolume” WHERE “State” IN (?, ?)) A, “Fileset” WHERE “ID” IN (?) AND “A”.“VolumeID” = “Fileset”.“VolumeID” ORDER BY “Fileset”.“Timestamp” took 0:00:00:00.000
Running Test took 0:00:00:00.611
No files examined, is the remote destination is empty?

2.0.3.6 was a few months in 2018 ahead of Log sql variables #3314 unfortunately, so SQL view is worse.

4

To me, it looks like the bug could also be here:

The variable tp is of type Tuple<string, object[]>, so an incorrect alignment of placeholders ? (generated by the code @ts678 linked to above) and arguments provided here could result in the string representation of tp.Item2 (Object[]) appearing in the query.

5

Ok, I think the problem is that the above code is calling ExecuteReader incorrectly. We either have to provide the parameter values (for the params Object[] parameterValues argument) separately, or together as an array. The code appears to be providing the first two arguments separately, and the remaining (tp.Item2) as an array.

..., RemoteVolumeState.Uploaded.ToString(), RemoteVolumeState.Verified.ToString(), tp.Item2)
2 Likes
6

Yes, both of you are right.

Something like that is making things go further indeed:

@@ -127,7 +127,11 @@ namespace Duplicati.Library.Main.Database
                 //First we select some filesets
                 var files = new List<RemoteVolume>();
                 var whereClause = string.IsNullOrEmpty(tp.Item1) ? " WHERE " : (" " + tp.Item1 + " AND ");
-                using(var rd = cmd.ExecuteReader(@"SELECT ""A"".""VolumeID"", ""A"".""Name"", ""A"".""Size"", ""A"".""Hash"", ""A"".""VerificationCount"" FROM (SELECT ""ID"" AS ""VolumeID"", ""Name"", ""Size"", ""Hash"", ""VerificationCount"" FROM ""Remotevolume"" WHERE ""State"" IN (?, ?)) A, ""Fileset"" " +  whereClause + @" ""A"".""VolumeID"" = ""Fileset"".""VolumeID"" ORDER BY ""Fileset"".""Timestamp"" ", RemoteVolumeState.Uploaded.ToString(), RemoteVolumeState.Verified.ToString(), tp.Item2))
+               var args = new List<object>();
+               args.Add(RemoteVolumeState.Uploaded.ToString());
+               args.Add(RemoteVolumeState.Verified.ToString());
+               args.AddRange(tp.Item2);
+                using(var rd = cmd.ExecuteReader(@"SELECT ""A"".""VolumeID"", ""A"".""Name"", ""A"".""Size"", ""A"".""Hash"", ""A"".""VerificationCount"" FROM (SELECT ""ID"" AS ""VolumeID"", ""Name"", ""Size"", ""Hash"", ""VerificationCount"" FROM ""Remotevolume"" WHERE ""State"" IN (?, ?)) A, ""Fileset"" " +  whereClause + @" ""A"".""VolumeID"" = ""Fileset"".""VolumeID"" ORDER BY ""Fileset"".""Timestamp"" ", args.ToArray()))
                     while (rd.Read())
                         files.Add(new RemoteVolume(rd));

it will need more research to check that it’s correct, of course.

2 Likes
7

done, I have sent a PR.

1 Like