Default_compressed_extensions.txt on linux system is case sensitive

asman · March 30, 2021, 9:51pm

Hi.
After long searching I found the reason for my performance problems. The default_compressed_extensions.txt does not work out of the box on Linux systems.

Seems the comparison is case sensitive, and all pics and videos with extensions xxx.MOV, yyy.JPG, zzz.MP4 are compressed - because only .mov .jpg .mp4 are mentionen in the default_compressed_extensions.txt

Is there any good way to make this case insensitive, without doubling all the entries in the default_compressed_extensions.txt file? (and if I double it, special cases like .Jpg are not covered as well)

Don’t get me wrong - the default extensions file is really great and I appreciate it. Would just be good to handle all cases on Linux systems.

Xavron · March 31, 2021, 12:06am

Not seeing any case insensitivity for this in the docs or in the code.

It looks limited in the code to look at as they switch to other values and pass that around so it shouldn’t be able to be re-checked for sensitivity just anywhere in the code. Thus, I don’t think I missed anything.

The probable things looks to be to download and use source with your change, write the code into a setting and submit that, or use something like Libre Office or Microsoft Office / Word to far more quickly switch the cases and duplicate into the file with the case changes.

Also, maybe using the terminal to adjust all cases of the files but that would affect the backup so you would probably want to redo the backup fresh after this one.

Or use the filters in Duplicati but I didn’t check to see if this is also or not. Just throwing that in. It does have a text box as well which might be this for all I know.

From what I see that’s what you have to work with.

drwtsn32 · March 31, 2021, 12:11am

Good find and thanks for reporting. Are you able to open an issue on the Github page? This should be trivial to fix.

Xavron · March 31, 2021, 12:36am

It looks like a past issue report might already answer their opinion on it actually Case insensitive filtering · Issue #2145 · duplicati/duplicati · GitHub

Also according to that

(?i) case insensitive

But I do not see if that’s possible with default_compressed_extensions.txt. When I looked at it, it wasn’t with that intent. If it doesn’t (and offhand without looking at it again I would say it doesn’t) then maybe that might be an issue to create.

But, also if it works with the filters inside Duplicati which I believe its for then could just go that way instead.

They might be a bit iffy or stubborn about it. I can see them based on the linked issue just closing it but there’s a chance (I suppose).

drwtsn32 · March 31, 2021, 1:33am

If it is, then it’s probably the right solution!

warwickmm · March 31, 2021, 2:29am

It might be as simple as providing a case-insensitive IEqualityComparer to the Dictionary of hints here:

github.com

duplicati/duplicati/blob/master/Duplicati/Library/Main/Options.cs#L1842




/// <summary>

/// Gets a lookup table with compression hints, the key is the file extension with the leading period

/// </summary>

public IDictionary<string, CompressionHint> CompressionHints

{

    get

    {

        if (m_compressionHints == null)

        {

            var hints = new Dictionary<string, CompressionHint>(Library.Utility.Utility.ClientFilenameStringComparer);



            string file;

            if (!m_options.TryGetValue("compression-extension-file", out file))

                file = DEFAULT_COMPRESSED_EXTENSION_FILE;



            if (!string.IsNullOrEmpty(file) && System.IO.File.Exists(file))

                foreach (var _line in Library.Utility.Utility.ReadFileWithDefaultEncoding(file).Split('\n'))

                {

                    var line = _line.Trim();

                    var lix = line.IndexOf(' ');

Currently, we provide a case-sensitive comparer if the filesystem is determined to be case sensitive.

Xavron · March 31, 2021, 8:57pm

My bad if anyone got my earlier replies that I deleted lol. Looks like IEqualityComparer should also work but the following quick change also works here as a test.

var hints = new Dictionary<string, CompressionHint>(StringComparer.OrdinalIgnoreCase);

.MP4 is indeed then treated as “Noncompressible”. Without that, .MP4 is treated as “CompressionHint.Default;”

Also the test shows that a .txt file will return as CompressionHint.Default so it looks to be okay.

Interesting so Library.Utility.Utility.ClientFilenameStringComparer looks like there’s already code to handle this. Just need to change that so a setting can override it. What do you think?

github.com

duplicati/duplicati/blob/master/Duplicati/Library/Utility/Utility.cs#L914


      
              {

                  try

                  {

                      var v = MonoDisplayVersion;

                      if (v != null)

                      {

                          var regex = new Regex(@"\d+\.\d+(\.\d+)?(\.\d+)?");

                          var match = regex.Match(v);

                          if (match.Success)

                              return new Version(match.Value);

                      }

                  }

                  catch

                  {

                      // ignored

                  }

          

                  return new Version();

              }

          }

github.com

duplicati/duplicati/blob/master/Duplicati/Library/Utility/Utility.cs#L836


      
          public static string ByteArrayAsHexString(byte[] data)

          {

              return BitConverter.ToString(data).Replace("-", string.Empty);

          }

          

          /// <summary>

          /// Converts a hex string to a byte array

          /// </summary>

          /// <returns>The string as byte array.</returns>

          /// <param name="hex">The hex string</param>

          /// <param name="data">The parsed data</param>

          public static void HexStringAsByteArray(string hex, byte[] data)

          {

              for (var i = 0; i < hex.Length; i += 2)

                  data[i / 2] = Convert.ToByte(hex.Substring(i, 2), 16);

          }

          

          public static bool Which(string appname)

          {

              if (!Platform.IsClientPosix)

                  return false;

warwickmm · May 3, 2021, 5:10pm

github.com/duplicati/duplicati

Make compression hints case-insensitive

opened 05:10PM - 03 May 21 UTC

warwickmm

enhancement good first issue

- [x] I have searched open and closed issues for duplicates. - [x] I have sea…rched the [forum](https://forum.duplicati.com) for related topics. ---------------------------------------- ## Environment info - **Duplicati version**: 2.0.6.1 - **Operating system**: All - **Backend**: All ## Description The [default_compressed_extensions.txt](https://github.com/duplicati/duplicati/blob/beaf03562fdcf4425e962085bdf7175d6a465f49/Duplicati/Library/Main/default_compressed_extensions.txt) file contains extensions for files that should not be compressed. However, these are treated in a case-sensitive manner. While filenames in general should be treated this way for case-sensitive filesystems, it seems unlikely for the extension to be case-sensitive when used to indicate the file type. We should consider making the compression hints use a case-insensitive match when determining whether to compress a file or not. This can be done by providing `StringComparer.OrdinalIgnoreCase` as the `IEqualityComparer` to the `Dictionary` of hints here: https://github.com/duplicati/duplicati/blob/beaf03562fdcf4425e962085bdf7175d6a465f49/Duplicati/Library/Main/Options.cs#L1842