IMO the decompressor code will be nearly impossible to fix without a better problem reproducer, meaning please help with your two or three suspect files. Opening an issue on third party library https://github.com/adamhathcock/sharpcompress will probably go nowhere (for good reason) as simply an oddity that shows up in Duplicati every 6 years. Unless the problem is in Duplicati use, your trouble-causing files are probably the key, but we’d have to find which one causes problem, then ideally isolate it to a single Duplicati block (one of the files with the block hash as its name).
This assumes that there’s a data-dependent decompression problem, which isn’t proven yet, as nobody but you has the data, and even you see that some dblock files work fine (in test restore).
You might also have other trouble causing files in your backup that will break later upon reading.
Additionally, you can’t avoid compact forever, unless you plan on storage space growing forever.
Developer opinions are of course welcome, but above is my thinking on what needs to be done.
If all tabs are using the same refresh token, there could be some kind of race where multiple tabs request a new access token, using the same refresh token. In most cases this would cause the anti-token-theft to trigger and log you out.
I think I have an idea for why this is happening then, but I not sure what the solution would be though. Thanks for the detailed debugging.
It is using the built-in .NET zip library in 2.0.9.108 by default.
You can set --zip-compression-library=SharpCompress to select SharpCompress.
If the data is not confidential, and the decompress error is reproducible, I would be able to debug some further if you can share the file. I think a major reason that we have not seen this more often is that it is specific to the LZMA (de-)compressor which is not commonly used.
Hi, sorry I’ve been away on vacation so only now starting to try and pick up from where I left off. I’m happy to supply the 3 suspect files as long as there’s some way for me to double-check their content. I still have them all extracted into folders of randomly named files, but no idea what to do.
What I had suggested was to copy them somewhere for RecoveryTool to try to read through. Idea would be to narrow it down to one file because if you mean check for sensitive source, that’s hard.
The AFFECTED command can take the dblock names as argument rows in GUI Commandline to translate to source file paths, but that’s only for source blocks still in the backup, so worsens over time as versions are deleted by retention, and blocks become wasted space for compact to clean.
might mean share with the main Duplicati author by PM from their profile because it reduces risk. You could certainly view each decompressed block in the dblock with something, but what a task.
OK, happy to try the “affected” command but I’m a bit lost to what parameters to use - if it’s easier to simply use a command-prompt than the GUI, what do I need for the paraeters according to the docs:
Duplicati.CommandLine.exe affected [] []
Currently the 3 files are all on my desktop: downloaded *.zip.aes, decrypted *.zip and then extracted folder
I suggested GUI Commandline which (maybe) avoids your question. Please click my prior link.
Once you click Commandline, you change its command and change Commandline arguments.
That box usually wants one argument per line, without quotes, so put some dblock names in it.
I certainly don’t know all the files there. /etc/security/opasswd worried me initially, however supposedly it’s MD5 hashes of the old passwords to stop reuse, so maybe is safe enough.
On my system it’s an empty file (so reveals not even a hash). You can look at your usages.
If this output is from multiple dblock files, you still have the chance to narrow it down some.
One puzzling thing is that the suspect files seem to contain their desktop block extractions.
duplicati-b0d7db3c279cf46c6a1bc4042bf347e0b.dblock is one (here AND Pastebin).
Regardless, you can PM 3 suspect files to primary dev if you like, try to reduce to 1 file, etc.
Click that for details, and the need for 2.0.9.107 can be avoided by giving 2.0.9.108 --zip-compression-library=SharpCompress
Duplicati.CommandLine.RecoveryTool.exe recompress zip "C:\tmp\recompress.pre" "C:\tmp\recompress.post" --zip-compression-library=SharpCompress
Start with one file in the input folder, have an empty output folder, and see which files give some decompression failure. If none do, that’s a bigger problem…
There’s also (if you’re willing to install something):
C:\Users\administrator\Desktop\DP>"C:\Program Files\Duplicati 2\Duplicati.CommandLine.RecoveryTool.exe" recompress zip "duplicati-b0d0fee3d1f784621834ad1e22e18ec47.dblock.zip" "POST\duplicati-b0d0fee3d1f784621834ad1e22e18ec47.dblock.zip" --zip-compression-library=SharpCompress
Creating target folder: C:\Users\administrator\Desktop\DP\POST\duplicati-b0d0fee3d1f784621834ad1e22e18ec47.dblock.zip
Listing files on backend: file ...
The folder C:\Users\administrator\Desktop\DP\duplicati-b0d0fee3d1f784621834ad1e22e18ec47.dblock.zip does not exist
C:\Users\administrator\Desktop\DP>"C:\Program Files\Duplicati 2\Duplicati.CommandLine.RecoveryTool.exe" recompress zip ".\duplicati-b0d0fee3d1f784621834ad1e22e18ec47.dblock.zip" "POST\duplicati-b0d0fee3d1f784621834ad1e22e18ec47.dblock.zip" --zip-compression-library=SharpCompress
Listing files on backend: file ...
The folder C:\Users\administrator\Desktop\DP\duplicati-b0d0fee3d1f784621834ad1e22e18ec47.dblock.zip does not exist
C:\Users\administrator\Desktop\DP>"C:\Program Files\Duplicati 2\Duplicati.CommandLine.RecoveryTool.exe" recompress zip ".\duplicati-b0d0fee3d1f784621834ad1e22e18ec47.dblock" "POST\duplicati-b0d0fee3d1f784621834ad1e22e18ec47.dblock.zip" --zip-compression-library=SharpCompress
Listing files on backend: file ...
Found 3384 files at remote storage
Found 3384 files at the remote storage, but none that could be parsed
show many formats. The URL style is file://hostname/folder%20for%20backup
I’m using the easier style C:\folder for backup – basically just path to that folder.
suggests it thinks it’s file: storage, but
suggests that maybe you pointed to a file, not a folder with a file.
Ok, that’s clearer now. I placed just one of the files in the folder and ran the command - I’ve narrowed it down to just one file:
T:\DP>"C:\Program Files\Duplicati 2\Duplicati.CommandLine.RecoveryTool.exe" recompress zip file://T:\DP\PRE T:\DP\POST --zip-compression-library=SharpCompress
Listing files on backend: file ...
Found 1 files at remote storage
Found 1 files which belongs to backup with prefix duplicati
1/1: duplicati-b0d0fee3d1f784621834ad1e22e18ec47.dblock.zip - downloading (49.91 MB)... recompressing ... done!
Download complete, of 1 remote files, 1 were downloaded with 0 errors
T:\DP>"C:\Program Files\Duplicati 2\Duplicati.CommandLine.RecoveryTool.exe" recompress zip file://T:\DP\PRE T:\DP\POST --zip-compression-library=SharpCompress
Listing files on backend: file ...
Found 1 files at remote storage
Found 1 files which belongs to backup with prefix duplicati
1/1: duplicati-b0d7db3c279cf46c6a1bc4042bf347e0b.dblock.zip - downloading (49.98 MB)... recompressing ... error: SharpCompress.Compressors.LZMA.DataErrorException: Data Error
at SharpCompress.Compressors.LZMA.LzmaStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at Duplicati.Library.Utility.Utility.CopyStream(Stream source, Stream target, Boolean tryRewindSource, Byte[] buf)
at Duplicati.CommandLine.RecoveryTool.Recompress.Run(List`1 args, Dictionary`2 options, IFilter filter)
Download complete, of 1 remote files, 1 were downloaded with 1 errors
There were errors during recompress of remote backend files!
T:\DP>"C:\Program Files\Duplicati 2\Duplicati.CommandLine.RecoveryTool.exe" recompress zip file://T:\DP\PRE T:\DP\POST --zip-compression-library=SharpCompress
Listing files on backend: file ...
Found 1 files at remote storage
Found 1 files which belongs to backup with prefix duplicati
1/1: duplicati-b0eea475a30f1400790621f5b8d96fd1d.dblock.zip - downloading (49.85 MB)... recompressing ... done!
Download complete, of 1 remote files, 1 were downloaded with 0 errors
Nice find! You can run affected on that one and see if it looks better. Other one wasn’t that bad.
If happy, I suppose you can PM that to @kenkendk as suggested. Narrowing it to one file inside archive may take some work, as at least the recompress code doesn’t log much. Possibly going back to the broken compact situation could get a better view, but you’ve done a lot on this so far.
I’m not looking in the code much, but there might be a partially converted file around somewhere which would (by its partialness) suggest how far in the original list of files it got before its bad file.
If it kept going after bad file, there may be one less file in output. If it stopped, file would be short.
I’m not sure what the file name would be, but it’s probably the newest in POST folder if it exists…
I tried the recovery tool again to recompress the problem zip file, and checked in the POST folder. The zip it was creating contains 922 items whereas the original is 2982. If there was a way to get it to show the files as it processes them then I could pull out just that file.
This is getting into or beyond developer level territory. I’d probably just send both files to the dev.
For getting the compact back in operation, you can treat this one file like a corrupted file, so use Recovering by purging files in the GUI’s Commandline after deleting (keep a local copy) that file.
If you pull a block out of the dblock file, that will probably get a complaint from a DB Recreate, or Restore might also be offended by a block that the DB says is in the dblock isn’t actually present.
But I’m not sure, and it’s not a well-tested path compared to sacrificing a dblock and affecting the files whose data it contained. The list-broken-files will probably look a lot like the affected.
“Test archived files” command in WinRAR might be interesting, even though I guess we heard that extraction (or something) worked. Possibly it puts up with weirdnesses better? Maybe test warns.
SimpleZIP test idea didn’t pan out. I corrupted an LZMA compressed file, but all I got was it saying
I just posted another possible way to find out which file is bad, but it might also just be due to SharpCompress getting confused. Still, seeing if some other ZIP tool gives warning may help.
One nice thing about identifying the problem dblock is that there should be lots of ways to find Duplicati version from either the manifest file (which is a text file), or guess from timestamps.
One nice thing about having a standalone way to test dblock is that it can be run from 2.0.8.1 extracted into a folder somewhere. We can get more hints on whether something broke lately.
