First - relative newbie to Duplicati - While I lived through the first Crashplan apocalypse and moved to Crashplan SMB, their most recent move to a 90 day retention limit has got me thinking it’s time to say goodbye. I’ve started using Duplicati as my replacement, and want to say “thank you”!
As I learn it, I’ve managed to answer all my questions through existing posts/information online. One stumped me (and Google).
I have a weekly backup of a Windows partition through Macrium Reflect meant for disaster recovery, but wanted something more “continuous” for more granular file recovery and so I set up an hourly backup of the same Windows partition with Duplicati.
What I found was that I was getting hourly dlist files of ~15MB, but no corresponding dblock files. The “compare” of backups showed that some files were added, mostly files being modified, and some files being deleted.
While it can’t explain the lack of dblock(s) when files are added, I assumed that the “modified” files were actually just changes in files’ time stamps rather than changes to the files’ data. I also wonder if, for the added files, if the size is 0, the dlist covers it without the need for a dblock.
Without a dblock, there would be no dindex for it, so just a dlist should find job Complete log showing:
Does it? Might as well verify that. Meanwhile I’ll get a little ahead.
Partly, but I’m not sure about completely.
0 length file content doesn’t go in dblock. Doing so was causing issues, so that changed a long time ago.
Time stamp changes are ordinarily recorded in a block of data that holds such information, and this goes into a dblock file along with other changes to files’ data and metadata (e.g. timestamps and permissions).
There is an open issue about a backup consisting only of timestamp changes not causing a dlist, but it seems inapplicable here because you got a dlist, plus you have info showing things like file deletions…
I think file deletion usually update the timestamp on its containing folder, so the time should be backed up. Seems to me like you should have gotten a dblock and dindex, but maybe there’s some way to avoid that.
I suspect that part of what was/is going on was that, since this was a new once an hour backup set (of a Windows 10 C: drive) going to a paid cloud provider, I’ve been actively watching which files are triggering backups.
I’ve been constantly adding to the exclude list as I find things that are changing (and causing a new version) but that I really don’t need to back up.
I suspect that that it was only after some such tweaking that I saw/see dlists without dblocks. If all of the files that are in a given dblock later land on the exclude list, can those dblocks get removed as part of Duplicati’s normal processing?
As an aside, these dlists are on the order of 10-15MB when I think I have proof that the changes in that backup version were to only a handful of files. From what I do know of dlists, 10-15MB seems large for some json to just describe those files, their timestamps, other meta data, etc. Makes me pretty sure there’s something about dlists that I don’t yet understand. What am I missing?
And again, thank you for your time. I am both a new user of Duplicati, so trying to get it working they way I want, but I’m also a software developer, so the internals interest me too!
That alone might get you a dlist without dblocks. The dlist would shrink due to those newly excluded files.
Your job log counts Added, Modified, and Deleted files. I think the new exclude would count as a deletion, however because it was done by exclude, it won’t modify its folder’s time (which would require a dblock).
This could happen eventually, if conditions call for that, when Compacting files at the backend takes place. Blocks from a file need to be kept as long as any backup version uses them. Retention policy adjusts that.
A manual purge is possible (but potentially accident-prone) if one really wants to purge all versions of a file.
Removal of dblock files is only done by compact which has several options if you want to try to tune them. Notes are at About → Show log → Live → Information or log-file=<path> and log-file-log-level=information.
If by “those files” you mean “a handful of files”, a dlist lists all files in a version, so should be fairly constant regardless of the amount of change in the source (which can result in varying amounts of dblock uploads).
How the backup process works and you can certainly play with an unencrypted backup to look in the .zip. Encrypted backups would needdecryption with AES Crypt (GUI) or Duplicati’s included SharAESCrypt.exe