ZIP64 seems unwilling to work in 2.1

Just reported on 2.1.0.5 on an issue closed in 2017, but I think there’s a new bug.

I first thought maybe it was due to using the built-in .NET .zip, but forcing it onto the SharpCompress library doesn’t help. Happens on Windows and Linux, and lack of the ZIP64 has been checked on a dlist file with both 7-Zip and in direct exam in hex viewer, with guidance from ZIP64 - Go Big Or Go Home.

Since Linux was the most recent text, here’s 2.0.8.1 in 7-Zip and hexdump -C:

$ 7z l *dlist*

7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,1 CPU Intel(R) Core(TM) i5-3470 CPU @ 3.20GHz (306A9),ASM,AES-NI)

Scanning the drive for archives:
1 file, 651 bytes (1 KiB)

Listing archive: duplicati-20250328T144832Z.dlist.zip

--
Path = duplicati-20250328T144832Z.dlist.zip
Type = zip
Physical Size = 651

   Date      Time    Attr         Size   Compressed  Name
------------------- ----- ------------ ------------  ------------------------
2025-03-28 10:48:32 .....          147          119  manifest
2025-03-28 10:48:32 .....           24           26  fileset
2025-03-28 10:48:32 .....          224          200  filelist.json
------------------- ----- ------------ ------------  ------------------------
2025-03-28 10:48:32                395          345  3 files
00000260  00 00 00 81 dc 00 00 00  66 69 6c 65 6c 69 73 74  |........filelist|
00000270  2e 6a 73 6f 6e 50 4b 05  06 00 00 00 00 03 00 03  |.jsonPK.........|
00000280  00 a6 00 00 00 cf 01 00  00 00 00                 |...........|
0000028b

Here’s 2.1.0.112:

Path = duplicati-20250328T150131Z.dlist.zip
Type = zip
Physical Size = 779
64-bit = +
00000290  00 00 00 00 00 00 00 81  03 01 00 00 66 69 6c 65  |............file|
000002a0  6c 69 73 74 2e 6a 73 6f  6e 50 4b 06 06 2c 00 00  |list.jsonPK..,..|
000002b0  00 00 00 00 00 00 00 2d  00 00 00 00 00 00 00 00  |.......-........|
000002c0  00 03 00 00 00 00 00 00  00 03 00 00 00 00 00 00  |................|
000002d0  00 a6 00 00 00 00 00 00  00 03 02 00 00 00 00 00  |................|
000002e0  00 50 4b 06 07 00 00 00  00 a9 02 00 00 00 00 00  |.PK.............|
000002f0  00 00 00 00 00 50 4b 05  06 00 00 00 00 03 00 03  |.....PK.........|
00000300  00 a6 00 00 00 03 02 00  00 00 00                 |...........|
0000030b

EDIT 1:

To attempt a similar repro without having to wait 4 days to backup millions of files:

This is 2.1.0.5 Stable on Windows 10, and control file is an over-4-GiB profiling log.

About → Show log:

Mar 28, 2025 11:59 AM: Failed while executing Backup "test 1" (id: 5)
System.NotSupportedException: Attempted to write a stream that is larger than 4GiB without setting the zip64 option
   at Duplicati.Library.Utility.Utility.CopyStream(Stream source, Stream target, Boolean tryRewindSource, Byte[] buf)
   at Duplicati.Library.Utility.Utility.CopyStream(Stream source, Stream target)
   at Duplicati.Library.Main.Volumes.FilesetVolumeWriter.AddControlFile(String localfile, CompressionHint hint, String filename)
   at Duplicati.Library.Main.Operation.Backup.UploadRealFilelist.<>c__DisplayClass1_0.<<Run>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at CoCoL.AutomationExtensions.RunTask[T](T channels, Func`2 method, Boolean catchRetiredExceptions)
   at Duplicati.Library.Main.Operation.BackupHandler.RunAsync(String[] sources, IFilter filter, CancellationToken token)
   at CoCoL.ChannelExtensions.WaitForTaskOrThrow(Task task)
   at Duplicati.Library.Main.Operation.BackupHandler.Run(String[] sources, IFilter filter, CancellationToken token)
   at Duplicati.Library.Main.Controller.<>c__DisplayClass17_0.<Backup>b__0(BackupResults result)
   at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action`1 method)
   at Duplicati.Library.Main.Controller.Backup(String[] inputsources, IFilter filter)
   at Duplicati.Server.Runner.Run(IRunnerData data, Boolean fromQueue)

EDIT 2:

There’s no dlist to look at, but the dindex doesn’t look like a ZIP64. Here’s the end:

Thanks for bringing this up. I have investigated a bit and it turns out that the new Zip compression (default) is capable of handling zip64 without any additional work.

SharpCompress is also able to do this, but for legacy reasons require explicit opt-in to enable zip64 support (in case there is an ancient tool that needs to read the zip file).

A bug introduced with the addition of the new zip library means that the flag is never actually set in SharpCompress, so any use of SharpCompress will be without zip64 support.

Ironically, setting the flag will cause the library to switch to SharpCompress, and leaving the flag alone will correctly support zip64.

I have addressed this in a new PR that deprecates the zip64 option, as all tools that work with zip files are expected to support zip64 extensions.

1 Like