Compare Feature Not Working

Rob_Roberts · August 2, 2022, 6:28pm

Hello everyone,

I am running Duplicati 2.0.6.3_beta_2021-06-17 on Ubuntu 22.04 LTS and I appear to be having an issue with the compare function. I have a backup with 12 versions and when I run the compare command by using

0
10
--full-result

I receive this output:

Listing changes
0: 08/02/2022 14:05:00
10: 08/02/2022 04:00:00

Size of backup 0: 0 bytes

No changes found
Size of backup 10: 0 bytes
Return code: 0

This does not make sense as there are changes between these backups. In addition it says the backups are 0 bytes which is not true. How can I resolve this so I may compare versions? I tried switching the 0 and 10 to see if that would make a difference and it did not.

JimboJones · August 2, 2022, 10:04pm

Welcome to the forums Rob_Roberts.

0 bytes is a bit of an ominous number of bytes to have or rather not have. A few questions off the top.

-Are you running the compare function in the GUI or via mono Duplicati.CommandLine.exe compare in a terminal?

-If you’re using the GUI does the job list show that you’ve backed up more than 0 bytes? The image below shows the details from my Sunday job showing the source data size along with the size of the backup and the amount of versions.

-Can you do a test restore from that backup set?

-Has the backup has been completing without errors (or warnings)?

Rob_Roberts · August 3, 2022, 12:22am

I am using the web GUI command line.
The GUI shows the following:

Source:
84.24 GB

Backup:
69.09 GB / 18 Versions (Now 18 versions but when I wrote the original post it was 12.)

I was able to do a successful test restore.
No errors or warnings in the log.

JimboJones · August 3, 2022, 2:32pm

Just to verify, when running the Commandline… you’re changing the dropdown to compare, removing anything in the Commandline arguments box then adding 0 10 --full-result each on it’s own line, leaving any advanced options as is they are then clicking on the Run "compare" command now button?

What if you run a compare with no Commandline arguments at all?

My other thought, it could be a Linux issue. I don’t have a 22.04 box kicking around at the moment but I do need a LAMP server for another job so I’ll try to get a few tests in while I’m working on that.

Rob_Roberts · August 3, 2022, 7:14pm

That is correct. I am going to the commandline GUI menu for the backup in question (the only backup I have configured), using the compare dropdown option, removing the existing text from the command line arguments, adding the input exactly as I pasted in the original post, not touching anything else and pressing the run compare button at the bottom.

The output of leaving the command line argument box empty and running compare is:

Finished!

            
Listing changes
  1: 08/03/2022 14:05:00
  0: 08/03/2022 15:05:00

Size of backup 1: 0 bytes


No changes found
Size of backup 0: 0 bytes
Return code: 0

JimboJones · August 3, 2022, 9:27pm

Got root?
Is Duplicati running as a service?

Also what’s changing? Any files being created/deleted, path changes or just modifications to an existing file data?

Rob_Roberts · August 3, 2022, 10:12pm

Duplicati is running as a service with root access. However please note the web GUI is firewalled to only be accessible from my IP.

If I add or delete a file to a folder being backed up, run a force backup and then compare the version 0 and 1 it will still show no difference when running the compare command. The file will still restore without any issue though so it does not appear to be an issue with the actual backup.

I also wanted to provide extra information about my setup. I am using an ARM based system and my backup has a custom prefix of “M”.

ts678 · August 4, 2022, 1:48am

As a long shot, The COMPARE command says it understands --exclude. Try un-checking any.

I don’t know if any other options might interfere. You could test a simple-as-possible test backup.

Rob_Roberts · August 4, 2022, 4:48am

I ran several test backups to try to narrow a specific cause to this issue. I have discovered it seems to happen when I have an excluded folder inside of a folder marked for backup. You can view my test backup config, with my Google Drive token removed, here: Duplicati Config - Pastebin.com.

wisobe · June 7, 2024, 11:19pm

Hi all ! Sorry to revive this topic, I’m having exactly the same result. Yesterday I was able to use the compare feature. Today I excluded a folder from my backup and it’s not working anymore. Is this the normal behaviour ? Can I do something about it ?

Thanks !

kenkendk · June 8, 2024, 6:55am

Hi @wisobe and welcome to the forum.

It has not been fixed afaik, so I have created an issue: Compare backup returns zero when a folder is excluded · Issue #5229 · duplicati/duplicati · GitHub

ts678 · June 8, 2024, 11:47am

Can you clarify? Was this exactly like the quote above yours (whose Pastebin details are now gone).

There are no steps to reproduce in the issue, just a pointer to this topic, so seeking some clarification.
Original post was on Linux command line. What is anyone else using? Works for me on Windows 10.
Although I use the GUI, looking at an Export As Command-line shows Source and exclude looking like:

"C:\Users\X\My Drive\\"
--exclude="C:\Users\X\My Drive\Duplicati Backups\\"

wisobe · June 10, 2024, 12:12am

Hi ! Using the GUI on a Raspberry Pi. Enclosed is a picture of an excluded folder.

Result is exactly as the original post.

ts678 · June 10, 2024, 12:38am

Thanks.Maybe it’s a Linux-only bug. Anyway, it doesn’t seem particular about what I exclude.
It can be a subfolder of an included folder as you did, or an unrelated folder, or a missing file.

kees-z · June 10, 2024, 11:30am

No, I’ve been affected by this bug on Windows for a very long time, this is a known bug for years: Remove Exclude Parameter If Running Compare via GUI Commandline - Support - Duplicati

Every time I was doing a COMPARE between 2 backup versions, I manually removed all EXCLUDE options from the list.
I’m surprised to find out that this issue doesn’t occur anymore in the latest beta, COMPARE operations work fine when I leave the EXCLUDE options in the list.

ts678 · June 10, 2024, 11:40am

My Linux test that had the issue was latest beta, 2.0.8.1. Issue sure seems erratic.

EDIT:

I am only testing 2.0.8.1 currently, but I tried to find a cross-platform bug, but didn’t.
Test is Source /tmp/short.txt with 1 letter, backup, change letter, backup, compare.
Adding in --exclude=random.txt breaks the compare on Linux, but not on Windows.
Differences are on Windows it’s C:\tmp, and notepad omits line end. Nano doesn’t.

Linux broken compare basically seems to think it has an empty backup, so outputs:

Listing changes
  1: 6/10/2024 8:28:24 AM
  0: 6/10/2024 8:30:07 AM

Size of backup 1: 0 bytes


  No changes found
Size of backup 0: 0 bytes
Return code: 0

Should be:

Listing changes
  1: 6/10/2024 8:28:24 AM
  0: 6/10/2024 8:30:07 AM

Size of backup 1: 2 bytes

  1 modified entries:
  ~ /tmp/short.txt


  Modified files:    1
Size of backup 0: 2 bytes
Return code: 0

Windows works even with --exclude=random.txt:

Listing changes
  1: 6/10/2024 8:46:39 AM
  0: 6/10/2024 8:47:11 AM

Size of backup 1: 1 bytes

  1 modified entries:
  ~ C:\tmp\short.txt


  Modified files:    1
Size of backup 0: 1 bytes
Return code: 0

Jojo-1000 · June 10, 2024, 1:45pm

The reason why it doesn’t work on linux for your example is the difference in filesystems, so different methods are used:

github.com

duplicati/duplicati/blob/bb7b3e5de41debedf222efbdc698878613ca2cb4/Duplicati/Library/Main/Database/LocalListChangesDatabase.cs#L151-L155


      
          else if (Library.Utility.Utility.IsFSCaseSensitive && filter is FilterExpression expression && expression.Type == Duplicati.Library.Utility.FilterType.Simple)

          {

              // File list based

              // unfortunately we cannot do this if the filesystem is case sensitive as

              // SQLite only supports ASCII compares

All the filtered paths are added to a table Filenames-XXX, which is then used in a where clause (at the bottom of this):

INSERT INTO "Previous-F574194ACDA6CB48885A8818019D565E" ("Path",
                                                         "FileHash",
                                                         "MetaHash",
                                                         "Size",
                                                         "Type")
SELECT "Path",
       "FileHash",
       "MetaHash",
       "Size",
       "Type"
FROM
  (SELECT "File"."Path" AS "Path",
          NULL AS "FileHash",
          "Blockset"."Fullhash" AS "MetaHash",
          -1 AS "Size",
          0 AS "Type",
          "FilesetEntry"."FilesetID" AS "FilesetID"
   FROM "File",
        "FilesetEntry",
        "Metadataset",
        "Blockset"
   WHERE "File"."ID" = "FilesetEntry"."FileID"
     AND "File"."BlocksetID" = -100
     AND "Metadataset"."ID"="File"."MetadataID"
     AND "Metadataset"."BlocksetID" = "Blockset"."ID"
   UNION SELECT "File"."Path" AS "Path",
                NULL AS "FileHash",
                "Blockset"."Fullhash" AS "MetaHash",
                -1 AS "Size",
                1 AS "Type",
                "FilesetEntry"."FilesetID" AS "FilesetID"
   FROM "File",
        "FilesetEntry",
        "Metadataset",
        "Blockset"
   WHERE "File"."ID" = "FilesetEntry"."FileID"
     AND "File"."BlocksetID" = -200
     AND "Metadataset"."ID"="File"."MetadataID"
     AND "Metadataset"."BlocksetID" = "Blockset"."ID"
   UNION SELECT "File"."Path" AS "Path",
                "FB"."FullHash" AS "FileHash",
                "MB"."Fullhash" AS "MetaHash",
                "FB"."Length" AS "Size",
                2 AS "Type",
                "FilesetEntry"."FilesetID" AS "FilesetID"
   FROM "File",
        "FilesetEntry",
        "Metadataset",
        "Blockset" MB,
        "Blockset" FB
   WHERE "File"."ID" = "FilesetEntry"."FileID"
     AND "File"."BlocksetID" >= 0
     AND "Metadataset"."ID"="File"."MetadataID"
     AND "Metadataset"."BlocksetID" = "MB"."ID"
     AND "File"."BlocksetID" = "FB"."ID" ) A
-- This assumes all filters are include filters:
WHERE "A"."FilesetID" = ?
  AND "A"."Path" IN 
    (SELECT DISTINCT "Path"
     FROM "Filenames-7B2A7B9904DC7743987C29221B815A75")

For the case-insensitive filesystems, this is instead done by iterating over all the results and applying the filter in C# rather than SQL.

Fix: #5232