Hello everyone, I’m new with using Duplicati. Is there a tool to compose custom backup retention rule or to verify it?
Hello and welcome!
I’m not aware of a tool, but if you tell us what you are looking for, we can provide the custom retention string and explain how it works.
My doubts are on how I can combine each place holder.
For example, I read somewhere a rule like this:
7D:0s,4W:0s,6M:1D,1Y:1M
Probably it’s not correct (or yes, I don’t know), but where can I read the meanings or the values of single tag? For example 0s, or maybe are there any other tags?
At least for the retention policy that you specify the allowed options are in a messy form direct from the code:
var valid_chars = /^((\d+[smhDWMY]|U):(\d+[smhDWMY]|U),?)+$/;
var valid_commas = /^(\d*\w:\d*\w,)\d\w:\d*\w$/;
example: Probably upper M is for month and and lower m is for minute. So s for second, m for minute, h for hour. Upper for larger periods of time. U for unlimited.
It should have validation that runs on it. For example you can’t enter only “1W” and have it work. It will tell you that it won’t. That looks like all there is.
It looks like you just need to look at it and figure out what you want. There’s no other UI or tool for it as far as I could see. Could use an ease of use improvement.
Check out my post here and let me know if it helps clear things up!
The two answers above go well together, and before writing a tool, somebody should update the manual.
Creating a new backup job explains the old simpler choices, and needs a section on new complex ones:
Fortunately when you get to Options screen, there’s a little bit of help-by-example, but U was added later.
Help text has caught up with U, but retention-policy has not. The builtin help text is better though, saying:
--retention-policy (String): Reduce number of versions by deleting old
intermediate backups
Use this option to reduce the number of versions that are kept with
increasing version age by deleting most of the old backups. The expected
format is a comma separated list of colon separated time frame and
interval pairs. For example the value "7D:0s,3M:1D,10Y:2M" means "For 7
day keep all backups, for 3 months keep one backup every day, for 10
years one backup every 2nd month and delete every backup older than
this.". This option also supports using the specifier "U" to indicate an
unlimited time interval.
but U can actually also be used on the time frame side (and is quite valuable there, because backups are removed when they are no longer in any time frame). For those who really want to see the code, it’s here, revealing that whoever posted the 0s interval might have been trying to get unlimited that way. Use U now.
Yep, since making that post I linked to above, I personally now use:
7D:U,3M:1D,2Y:1W,U:1M
Hello, yes your suggets are clear for me. Thanks
many thanks, I’m trying to choose the best retetion strategies depending on every type of backup and data that I have.
7D:U,3M:1D,2Y:1W,U:1M
If I well understood, the meaning is:
for last 7 days - all versions
for last 3 months - only one per day
for the last 2 years - only one per week
for ever - almost one per months
is it right?
sorry but perhaps I did some confusion in reply and posting.
I’m thinking about this retention
2W:U,6M:1D,1Y:1W,2Y:1M
Could it works correctly?
Yep, you got it right!
Yes, this policy would work fine. But note that once a backup is over 2 years old, it will be deleted.
ok clear, so if I modified a file 3 years ago (and I made a backup near that date), I can’t restore it now.
Yes, that’s correct. The retention is calculated by backup date, not by file modification time.
1 Like
You can restore version of 2 years ago if file was there then, but not a version in no time frame.
In your proposal, the longest time frame was 2Y, but you can make it whatever you like, even U.
Thanks @ts678, I misread the original question.
The question was kind of vague, so I thought I’d split it into cases. Do you think we could improve docs?
Maybe it will take a couple of people to try to come up with the right words, but manual now has nothing.
One other thing I’d like to put in is that units (e.g. day, month), don’t try to follow local timezone/calendar.
People sometimes get surprised by that, but to retention policy, time is just time, measured to seconds.
Hi there. I’d like to take a shot at documentation.
Could you please refer me to the locations in the sources …
- where the actual decision algorithm about retention is in the code?
- where online helptexts and manual entries are kept in the sources?
I have a good idea about how it works by analyzing my own logs. But I’d rather compare it with the actual “truth in the code”.
I took the text off the Web GUI and adapted it. Might be a little too long, so better suited for the manual?
Please let me know how i could contribute that or shortened versions.
Use this option to reduce the number of Backups that are kept with increasing Backup age by deleting most of the old Backups.
The expected format is a comma separated list of colon separated Timeframe:Interval pairs.
The Timeframe part is interpreted as the maximum age of Backups to be considered per pair.
The Interval part is the minimum age difference between two consecutive Backups to be retained.
Take “7D:0s,3M:1D,10Y:2M” as an example.
The retention algorithm will sort the list by Timeframe length and walk through it from the shortest to the longest Timeframe.
This means:
“Retain all (Interval ‘0s’) Backups up to an age of 7 days (Timeframe ‘7D’).
From Backups older than 7 days up to the age of 3 months (‘3M’) retain only those that differ at least 24 hours (‘1D’) in age.
Then retain Backups spaced 2 months apart from each other for the Backups aged between 3 months and 10 yeras.
Finally, discard all backups older than 10 years.”
You may use “U” as the longest (unlimited) Timeframe, e.g. “U:1Y”, to keep certain Backups forever.
Accepted Frames/Intervals are (s, m, h, D, W, M, Y) for (seconds, minutes, hours, Days, Weeks, Months, Years) respectively.
You should not specify two pairs with equal Timeframe value.
Intervals only make sense, if they are increasing along with increasing Timeframe sizes.
Welcome to the forum @dennis-mb
Sounds great to me. Thank you!
https://github.com/kees-z/DuplicatiDocs/blob/master/docs/03-using-the-graphical-user-interface.md
https://duplicati.readthedocs.io/en/latest/ gives the preferred way of getting changes done to manual:
Corrections/changes/additions may be submitted as a pull request at Github: GitHub - kees-z/DuplicatiDocs: Duplicati User's Manual
Duplicati code and text also works by pull requests which can be made here.
That’s the writing challenge – trying to explain this clearly enough without going too long on the length…
You can look at the short and long per-option help texts. Even long ones are sometimes not super long.