I use veracrypt to backup some stuffs. I don’t crypt a whole USB HD\pen drive but use files and mount them. Of course I wish to backup them =)
When I create volumes veracrypt allow me to create sparse files, now according to some test which I performed, duplicati can’t distinguish sparse or not sparse files: it backup always the whole file. This is correct? In order to obtain a greater reliability what type of file is preferable, sparse or ordinary?
I’m talking about NTFS scenarios, windows 10-11 platform.
I think it just does sequential file read, and the OS returns a sparse file just like a non-sparse.
Regardless, the backup represents a file as an ordered list of blocks (referred to as blocklist).
Deduplication (which is done to all block data) allows reuse of blocks of NUL from sparse file.
Restore of sparse file will no longer be sparse, unless the new Canary restore rewrite does it.
I rather doubt it does, but if it does, there’s no info to guarantee perfect match of sparse area.
At least that will avoid surprise expansion on restore – by using the larger files from the start.
Beyond that, I expect OS-level support of sparse files has likely been well-proven as reliable.
From what I can read, Veracrypt uses NTFS sparse files. This works by not allocating actual space before it is used.
For example, you create a 200GiB file, but write 1MiB. The file on-disk is only 1MiB large (or something close to that). If you write more data, the on-disk file starts to grow.
Duplicati does not understand sparse files (or NTFS) so it simply reads the file. All the data that was never used is just returned as zeros to Duplicati, so Duplicati sees a 200GiB file. It will unfortunately read and process the entire 200GiB file, which might take some time.
Fortunately, Duplicati uses de-duplication so it only stores a single block of zeroes, so the backup size will more closely reflect the on-disk size.
The problem comes when restoring. Since there is no knowledge of the sparse file, Duplicati will restore the full 200GiB (including zeroes) to disk. If you have plenty space, it should not cause problems, but you cannot get the sparse format back after a restore.