Purge Inter-month Incremental Backups

Features
1

Hi - looking to move my backup process to cloud based vs local external drive. I currently run a linux machine and have used Back In Time quite successfully for many years. The feature I really enjoy in BIT is the smart purge that allows me to keep incremental backup like:
6 per week
1 per week/month (4)
1 per month/year (12)
1 per year for each previous year
This, of course, keeps my back up space to a minimum.

I see Duplicati has some auto purge features, but selecting “older than X” would mean missing out on getting to a file that’s older than “X” and newer than the initial backup. (Yes, I know something that was transient within a month will also be lost with BIT.)

So, wondering if I can do a manual version. For instance, at the start of a new month, manually batch select all the incremental backups for the previous month up the the last one, and delete them? That way, I’d have 1 incremental for each month. At the end of the first year, my backups would show 13 instances. The initial and an incremental for each month.

It doesn’t seem like this would break any continuity between the remaining monthly incremental backup. Will this work?
Thanks!

2

New retention policy deletes old backups in a smart way announced this, but seemingly the manual hasn’t caught up. Here’s a screenshot example of something that sounds an awful lot like what you’re asking for:

image
image.png1259×326 30.4 KB

1D:U,1W:1D,4W:1W,12M:1M is retention rules that progressively thin out backups depending on their age.

  • For 1 day, keep all
  • For 1 week, keep 1 per day
  • For 4 weeks, keep 1 per week
  • For 12 months, keep 1 per month
  • After 12 months, keep no backup

Your preferences differ, but your example near the top of the post is roughly 1W:1D,4W:1W,12M:1M,U:1Y

You know about the transient file issue. One additional caveat here is there’s no calendar. It’s all intervals.

C:\Program Files\Duplicati 2>Duplicati.CommandLine.exe help retention-policy
  --retention-policy (String): Reduce number of versions by deleting old
    intermediate backups
    Use this option to reduce the number of versions that are kept with
    increasing version age by deleting most of the old backups. The expected
    format is a comma separated list of colon separated time frame and
    interval pairs. For example the value "7D:0s,3M:1D,10Y:2M" means "For 7
    day keep all backups, for 3 months keep one backup every day, for 10
    years one backup every 2nd month and delete every backup older than
    this.". This option also supports using the specifier "U" to indicate an
    unlimited time interval.

C:\Program Files\Duplicati 2>
3

@ts678 - thank you. Sounds like just what I was looking for. Now wondering how I merge my old BIT backups with Duplicati backups on Backblaze B2?

Is there a way to use Duplicati to encrypt my local backups and simply push them to Backblaze in case I ever need them? Maybe there’s a better path I should travel. Thoughts…

4

I’m not a Back in Time expert, but research suggests that you have a hard drive with filesystem of backup trees of unencrypted files, with space usage reduced through using hard links between identical copies of some file. Access to any file could be done right from Linux, if desired (something that Duplicati can’t do).

If that’s correct, if you just pointed Duplicati to your top level directry, it could probably ingest it slowly, with identical files getting completely deduplicated, and similar files (e.g. a file appending situation) getting less deduplication. You might wind up with a BIG file list with similar names below, and quite a lot of storage to keep all versions’ data blocks. Duplicati does allow thinning of versions over time through retention policy, but any file that exists only between retained versions has no backup (as discussed above by both of us).

Alternatively, you could use any Linux archiver that can put a big tree of small files into one enormous file, but the trick would be to find one that knows the hard link trick or can deduplicate really well another way.

8.3.3 Hard Links gives me hope that tar can handle them. A compressor can then squeeze size further, and an encryptor such as AES Crypt can make an encrypted version to upload in case you lose the drive which perhaps you’ll keep for easy access to older files? I forget whether tar does random access well, however either compression or encryption will probably make random access to older files very awkward.

I don’t know how much importance you place on keeping old versions, or backups at all, but best practice calls for multiple backups done different ways. Duplicati (still in beta) does fail on rare occasions, and old backup stored somewhat separate from your Duplicati new backup would be a good fail-safe to preserve.

Duplicati also gets somewhat stressed and slow if backups get huge, so that’s another factor to consider. For example putting your old backup into the same Duplicati backup as the new one worsens both issues. Beyond that, initial upload can take a long time if network is slow. Uploads in small increments works best.

“better path” kind of depends on your priorities. Feel free to clarify, otherwise there’s some general advice.