Custom backup retention testing

Support
1

I have created two backup jobs.

  1. Job backups daily with a Custom backup retention of 7D:1D,4W:1W,12M:1M,10Y:1Y

So this keeps one backup for each …
… day of the last 7 days
… week of the last 4 weeks
… month of the last 12 months
… year of the last 10 years

Is this correct?

  1. Job backups weekly (on mondays) with a Custom backup retention of 4W:1W,12M:1M,10Y:1Y

So this keeps one backup for each …
… week of the last 4 weeks
… month of the last 12 months
… year of the last 10 years

Is this correct?

Is there a way to test this, so that I can be sure, it will work the way I want it to?

2

Welcome to the forum @johnD

Your understanding looks correct, but I just want to clarify one thing - the timeframes overlap. So in your first example, you get 1 backup per day for one week, followed by 1 backup per week for the timeframe of 2 weeks ago through 4 weeks ago.

I don’t know if it will help, but I tried to give a very detailed description of the behavior here:

L

3

@Lerrissirrel Thanks for your reply. I think I got it now.

So for example 1, it should be 7D:1D,3W:1W,11M:1M,9Y:1Y to get:
… for 7D:1D: 2023-05-11, 2023-05-10, 2023-05-09, 2023-05-08, 2023-05-07, 2023-05-06, 2023-05-05
… for 3W:1W: 2023-04-28, 2023-04-21, 2023-04-14
… for 11M:1M: 2023-03-14, 2023-02-14, 2023-01-14, 2022-12-14, 2022-11-14, 2022-10-14, 2022-09-14, 2022-08-14, 2022-07-14, 2022-06-14, 2022-05-14
… for 9Y:1Y: 2021-05-14, 2020-05-14, 2019-05-14, 2018-05-14, 2017-05-14, 2016-05-14, 2015-05-14, 2014-05-14, 2013-05-14

And for example 2, it should be 4W:1W,11M:1M,9Y:1Y to get:
… for 3W:1W: 2023-05-05, 2023-04-28, 2023-04-21, 2023-04-14
… for 11M:1M: 2023-03-14, 2023-02-14, 2023-01-14, 2022-12-14, 2022-11-14, 2022-10-14, 2022-09-14, 2022-08-14, 2022-07-14, 2022-06-14, 2022-05-14
… for 9Y:1Y: 2021-05-14, 2020-05-14, 2019-05-14, 2018-05-14, 2017-05-14, 2016-05-14, 2015-05-14, 2014-05-14, 2013-05-14

Right?

4

I think you’ve got it, but I need to double check first - Is the “3W:1W” a typo here, and you meant “4W:1W”?:

In your initial example you used 4W:1W, and the result that you describe, 4-28, 4-21, 4-14 is, essentially, correct for 4W:1W since you have a 7D:1D. I say “essentially” due to what I describe two paragraphs down.

So I either confused you, or that’s just a typo.

The piece that I can’t confirm or deny is exactly WHICH backups will get saved when your frequency of existing backup is greater than your retention policy. For instance, your example calls out specific individual days’ backups one per week after the first week. I do not know if 4-28, 4-21 and 4-14 would be the ones retained. I don’t know the internal algorithm.

Note the generic way I described that; you’ll be in a similar situation when Duplicati has to figure out which of the “1 per week” backups it will keep to satisfy the “1 per month” piece of the policy. In addition, this also applies to even the “7D:1D” if you’re taking more than one backup a day!

Hopefully this helps and I’m not confusing you more!

Mark

5

I’m pretty sure dates shown are made up, but job backups weekly on Mondays doesn’t likely fit them.
Retention doesn’t make old backups, but can only remove them. It’s also based on time not calendar, which also leads to oddities like a daily backup moving around slightly and sometimes getting deleted because it ran a bit less than 1D after previous backup. Avoid issue by using 23h or similar instead…

worries me more. In the overlapping timeframe design, this one starts where the 11M timeframe ends.
A log-file at log-file-log-level=information has highlights of the work (profiling has details), for example:

2023-05-11 17:57:47 -04 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-StartCheck]: Start checking if backups can be removed
2023-05-11 17:57:47 -04 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-FramesAndIntervals]: Time frames and intervals pairs: 1.00:00:00 / Keep all, 14.00:00:00 / 08:00:00, 28.00:00:00 / 7.00:00:00, 365.00:00:00 / 31.00:00:00
2023-05-11 17:57:47 -04 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-BackupList]: Backups to consider: 5/11/2023 4:50:02 PM, 5/11/2023 3:50:02 PM, 5/11/2023 2:50:02 PM, 5/11/2023 1:50:02 PM, 5/11/2023 12:50:02 PM, 5/11/2023 11:50:06 AM, 5/11/2023 10:50:02 AM, 5/11/2023 9:50:35 AM, 5/11/2023 8:50:02 AM, 5/11/2023 7:50:02 AM, 5/11/2023 6:59:29 AM, 5/11/2023 6:39:29 AM, 5/10/2023 8:50:03 PM, 5/10/2023 7:50:03 PM, 5/10/2023 6:50:06 PM, 5/10/2023 5:50:02 PM, 5/10/2023 4:50:00 PM, 5/9/2023 3:50:06 PM, 5/9/2023 6:59:08 AM, 5/8/2023 3:50:02 PM, 5/8/2023 7:00:57 AM, 5/7/2023 7:02:31 AM, 5/6/2023 4:50:02 PM, 5/6/2023 7:50:03 AM, 5/5/2023 3:50:02 PM, 5/5/2023 6:57:32 AM, 5/4/2023 3:50:03 PM, 5/4/2023 6:57:53 AM, 5/3/2023 5:50:03 PM, 5/3/2023 9:50:03 AM, 5/3/2023 12:51:14 AM, 5/2/2023 3:50:05 PM, 5/2/2023 6:57:15 AM, 5/1/2023 2:50:23 PM, 5/1/2023 6:50:00 AM, 4/30/2023 3:50:08 PM, 4/30/2023 7:06:10 AM, 4/29/2023 2:50:12 PM, 4/29/2023 6:50:03 AM, 4/28/2023 2:50:00 PM, 4/28/2023 5:51:09 AM, 4/27/2023 9:50:50 PM, 4/24/2023 6:45:05 AM, 4/16/2023 2:50:06 PM, 4/2/2023 12:50:05 AM, 2/25/2023 7:06:49 AM, 2/2/2023 4:34:18 PM, 1/19/2023 7:19:19 AM, 12/14/2022 7:08:47 AM, 11/29/2022 7:50:04 PM, 11/29/2022 4:50:01 PM, 11/29/2022 1:50:04 PM, 11/25/2022 2:50:01 PM, 11/8/2022 6:35:03 AM, 10/3/2022 6:28:06 AM, 8/27/2022 6:05:27 AM, 7/21/2022 7:50:03 AM, 7/20/2022 4:50:07 PM, 7/20/2022 6:50:06 AM, 6/13/2022 3:50:00 PM, 5/31/2022 6:50:03 PM, 5/23/2022 2:09:32 PM
2023-05-11 17:57:47 -04 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-BackupsToDelete]: Backups outside of all time frames and thus getting deleted:
2023-05-11 17:57:47 -04 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-AllBackupsToDelete]: All backups to delete: 5/10/2023 5:50:02 PM

6

@Lerrissirrel Oh ya … I am really confused now! @ts678 same here.

I’ll try to describe what I exactly want in words.

Here is a quote from the docs (Using the Graphical User Interface - Duplicati 2 User's Manual)

Smart backup retention: retains one backup for each of the last 7 days, each of the last 4 weeks, and each of the last 12 months.

Essentially this is what I want PLUS one backup for each of the last 10 years.

So for Job #1 where a daily backup is created, it should retain …

  • one backup for each day of the last 7 days (7 backups)
  • one backup for each week of the last 4 weeks (except for the current, 3 backups)
  • one backup for each month of the last 12 months (except for the current, 11 backups)
  • one backup for each year of the last 10 years (except for the current, 9 backups)

So in total I would have 30 backups if it would run longer than 10 years.

How would my Custom backup retention look like?

And for Job #2 where a weekly backup (on mondays) is created, it should retain …

  • one backup for each week of the last 4 weeks (4 backups)
  • one backup for each month of the last 12 months (except for the current, 11 backups)
  • one backup for each year of the last 10 years (except for the current, 9 backups)

So in total I would have 24 backups if it would run longer than 10 years.

How would my Custom backup retention look like?

7

These look better than the example with the specific dates. Do you understand the issues there, e.g.

  • Dates claimed to be mondays were not.
  • 9Y age cutoff had 2013-05-14, which exceeds 9 years. Read the manual to see how this works.
  • Things like 11M seemed like maybe it thought it followed the 4W before it. Time frames overlap. They’re an age. Smart retention is like custom retention 1W:1D,4W:1W,12M:1M with time frames being 1W and 4W and 12M. The 0-4 week minimum interval rule only applies after 1 week age, without you having to make it 3W. The 0-1 week age got covered by the 1W time frame interval.

You probably weren’t looking at a calendar. Duplicati doesn’t either. An interval is counted in seconds, meaning 1D is 24h is 1440m is 86400s, so if run time on daily backup is a bit off, 86401s will delete it.
Daily backup with exactly-a-day interval invites unwanted deletes. Unfortunately it’s a popular setup…

Then extend the 1W:1D,4W:1W,12M:1M with ,10Y:1Y

8

OK. So the Custom backup retention strings in my initial post were correct.

What confused me a lot was this:

Does “followed by 1 backup per week for the timeframe of 2 weeks ago through 4 weeks ago” make sense?

And then I confused you with my dates …

So … much ado about nothing :wink:

9

It makes sense if you take the 2 as a typo (or logic error) for 1 which is where smallest timeframe ends.
The 4W has 1W (a.k.a. 7D) of its youngest age carved out for a different interval (1D), leaving it weeks ending at 2W, 3W and 4W for its 1W interval. Week ending at 2W ago began at 1W ago though (logic).