Confirming my understanding of backup retention

1

Hi Guys,

Thanks so much for the program, it seems like exactly what I need to do my backups.

I’m looking for confirmation that I’ve set Duplicati up correctly for my needs.

My usage is this:
• Store all my photography RAW files on the B2 cloud, automatically as I transfer them from cards onto my computer HDD. I don’t want these ever deleted from B2.
• Store edits of files made within these folders and additional files created automatically. Ie. I ed it a photo and save the .PSD file into the folder with the RAW’s so I know where to find it, I want that file backed up. (I only want one copy of it though, the most recent edit) I always want a copy of these files to exist on B2 no matter what happens to my PC, but I don’t want 10 versions of this file stored in the cloud.

I have setup my backup to run nightly through Duplicati, with the following settings:
• AES 256 encryption
• B2 cloud storage setup correctly
• Source Data identified correctly
• Automatically run backup daily once a day.
• Keep all Backups

It’s the last one that I’m not sure about. If I don’t make changes to the files will it keep uploading them over and over again?

Can someone confirm to me that I’ve got the settings right for my usage case?

Cheers,
Dan

2

Hi @danielc and welcome to the forum! I’m glad you like Duplicati. It’s got a lot of options, with some limits.

The “keep uploading” isn’t an issue because Duplicati only uploads file data not previously backed up, and otherwise it just references previous file blocks when it creates a new version of its list-of-all-files-right-now.

If that’s enough, you might be good to go, however there were other wishes and maybe some optimizations.

Limiting backup of PSD files to the latest version likely requires another job with different retention settings.

Filters would steer RAW files one way, and PSD files the other. I don’t use filters, but I did find forum issues.

Making PSD files age off exactly as wished might be hard if you also want to keep the latest version forever, even if you delete it (which might be a reasonable precaution for RAW files). You can use retention-policy or something similar to cut old versions, but if you delete a PSD, and don’t notice soon enough, it may age off.

My comment about optimization has to do with whether a file is compressible. If already compressed, there’s little point in trying further, so Duplicati has a compression-extension-file option (and default file) for control.

Depending on how a PSD file is built, you might also defeat Duplicati’s block-based deduplication attempts, but that just means an edit you do might result in a bigger upload than an easy case like a data file append.

EDIT: If nobody else can think of how to solve the keep-the-final-PSD-but-not-the-drafts problem, maybe it could be solved through your using different folders for files-to-keep-forever versus the intermediate work, which would also bypass any remaining filter issues if there are any. You can search to read current status.

3

I agree with @ts678, if you’re looking for different retention rules based on file type then you’ll need two different backup jobs.

The “Keep all versions” option sounds like what you want for your RAW files since they never get edited directly (right?) but for the PSDs you’d want something like “Keep a specific number of backups” set to 1 (aka --keep-versions=1) meaning once there are more than 1 versions, the oldest ones are deleted.

Note that this may NOT keep a file in your backup if it’s deleted from the source - though I’m not sure if that’s something you need or not.

4

Confirming that --keep-versions=1 does not retain any backup of the file if it is deleted and then backup done.

Duplicati versions refer to the view of the entire set of files at some time. This works well if one has to recover back to a particular time, whether for one file or many, but it doesn’t limit the versions separately for every file.

Some backup programs go the other way. You get versions per-file, but point-in-time restores become harder.