# New retention policy deletes old backups in a smart way **URL:** \ **Category:** Announcements\ **Created:** [January 23, 2018, 6:47pm UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195 "2018-01-23T18:47:03Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![basje](https://forum.duplicati.com/user_avatar/forum.duplicati.com/basje/32/914_2.png) [@basje](https://forum.duplicati.com/u/basje)\ **Post date:** [January 24, 2018, 8:06pm UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195/21 "2018-01-24T20:06:03Z")
> [@Tekki](#): > > > [@basje](#): > > > > So what happens to the --retention-policy option? I currently have this: > > > > –retention-policy=6W:0s,16W:1W,2Y:1M > > > > and > > > > “Keep this number of backups”=“A specific number”=40 > > hm … I haven’t tried this, but at the moment there _might_ actually be situations where you’d end up with **less** then the specified 40 versions since the “Keep this number of backups” option doesn’t take into account, that the “Retention Policy” will already delete some backups. > > So even if the GUI might prevent having multiple Backup retention options simultaneously in the future, I’ll still try to have a look at code for combining these options soon-ish. If it really turns out to be a problem, then I’ll try to change it so that the “Keep this number of backups” is taken into account **after** both the “Retention Policy” and the “Delete backups that are older than” options were applied. For me the 40 backups set via the GUI policy will be just fine, and I don’t mind that this will be the limit and more will be deleted, regardeless of any custom policy set via the advanced command line options. I was just wondering whether the command-line switch will keep working? Could it be possible to set both an “custom policy” via the advanced command line options and via the GUI? Because that should not be possible? I tried it but then the GUI seems unable to determine the correct policy and I need to remove all policies and clear cache to fix it. ---
**Author:** ![kees-z](https://forum.duplicati.com/user_avatar/forum.duplicati.com/kees-z/32/29_2.png) [@kees-z](https://forum.duplicati.com/u/kees-z)\ **Post date:** [January 24, 2018, 8:39pm UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195/22 "2018-01-24T20:39:55Z")
Nice to have the `--retention-policy` option integrated in the GUI, but I guess at least the `--keep-time` option should stay available somehow when choosing “Smart backup retention” or “Custom backup retention”. The retention policy for Smart retention is “1W:1D,4W:1W,12M:1M”. This leaves all backups older than 12 months untouched. Combined with for example `--keep-time=3Y`, all versions older than 3 years will be deleted. As a result, backup versions will not continue to increase indefinitely. In the current version, you have to choose between smart/custom backup retention and keep time/versions, preventing deletion of old backup versions from the GUI. ---
**Author:** ![JonMikelV](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/j/439d5e/32.png) [@JonMikelV](https://forum.duplicati.com/u/JonMikelV)\ **Post date:** [January 25, 2018, 12:39am UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195/23 "2018-01-25T00:39:46Z")
Ahhh - so “1W:1D,4W:1W,12M:1M” combined with `--keep-time=3Y` means that as the 12 months worth of monthly backups age into 13+ months they’ll just sit around as monthly backups until they become 3 years old at which point they’ll be deleted. Perfect for my needs! ---
**Author:** ![lseg](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/l/ac8455/32.png) [@lseg](https://forum.duplicati.com/u/lseg)\ **Post date:** [January 25, 2018, 7:23pm UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195/24 "2018-01-25T19:23:44Z")
When will we see this in an “official” release? I am using 2.0.2.1, I’d prefer not to use a Canary stream, which I understand is a developpers mainline. 2.0.2.1 is still in Beta Phase, so I expect the canary stream to be alpha or even lower? ---
**Author:** ![kenkendk](https://forum.duplicati.com/user_avatar/forum.duplicati.com/kenkendk/32/5305_2.png) [@kenkendk](https://forum.duplicati.com/u/kenkendk)\ **Post date:** [January 26, 2018, 8:51am UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195/25 "2018-01-26T08:51:04Z")
> [@Tapio](#): > > EDIT: does it mix old setting and new setting in the same command line? UI displays “Keep specific number: 3”, the saved command line is: > > -keep-versions=3 --retention-policy=“8D:2D,4W:1W,12M:1M” > > In another backup I have: “–keep-time=12M --retention-policy=“7D:1D,4W:1W,60M:6M”” > > Seems like a mess. Seems as if the UI does not very well save new settings. If you use the dropdown to apply the settings, it should clear the other (i.e. use either `--keep-time` or `--retention-policy`). If you use the Advanced options, it just applies those _after_ the dropdown options. ---
**Author:** ![Tapio](https://forum.duplicati.com/user_avatar/forum.duplicati.com/tapio/32/3371_2.png) [@Tapio](https://forum.duplicati.com/u/Tapio)\ **Post date:** [January 27, 2018, 12:57pm UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195/26 "2018-01-27T12:57:15Z")
No, it does not clear anything… I use the basic dropdown, set custom retention policy, and after save and re-visiting it again presents me keeptime=12M. I then used the advanced options, only added custom retention. After save and revisit, Duplicati show the custom retention in (5) General options and advanced options. But when I look into “Export configuration” I can see it has stored both, Keep-time and retention-policy. ---
**Author:** ![drakar2007](https://forum.duplicati.com/user_avatar/forum.duplicati.com/drakar2007/32/230_2.png) [@drakar2007](https://forum.duplicati.com/u/drakar2007)\ **Post date:** [January 30, 2018, 5:05pm UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195/27 "2018-01-30T17:05:51Z")
FYI, i’ve found the canary builds to be very stable. If you were hesitating in installing a Canary version, I’d say it’s unnecessary - even if you just want to install the current one to test the current features and then wait until major releases later. ---
**Author:** ![Pectojin](https://forum.duplicati.com/user_avatar/forum.duplicati.com/pectojin/32/1231_2.png) [@Pectojin](https://forum.duplicati.com/u/Pectojin)\ **Post date:** [February 2, 2018, 6:36pm UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195/28 "2018-02-02T18:36:48Z")
It’s actually my experience that canary has the least problematic bugs, so being on canary _might_ be safer than beta right now 😉 Of course the problem with canary is that it’s not been tested by more than a few people before it’s released, so that assumption could change any time. ---
**Author:** ![JonMikelV](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/j/439d5e/32.png) [@JonMikelV](https://forum.duplicati.com/u/JonMikelV)\ **Post date:** [February 2, 2018, 9:47pm UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195/29 "2018-02-02T21:47:13Z")
Since the order of code release is `canary` → `experimental` → `beta` → `stable` you might want to give `experimental` a try. This post includes a quick summary of the different release channels: > [@Is there any approximate release date for the next beta?](https://forum.duplicati.com/t/is-there-any-approximate-release-date-for-the-next-beta/2147/2/): > > Unfortunately, as an open source & volunteer project it is very difficult to estimate how much time people will be able to donate to working on Duplicati. As a result of that absolute dates for release are pretty much impossible - though we do get hints from time to time when kenkendk posts comments saying a new experimental is coming “soon” or “this month”. The different channels are described more fully in the link below, but the short answer is: stable: no known major (and few if any min… ---
**Author:** ![Pectojin](https://forum.duplicati.com/user_avatar/forum.duplicati.com/pectojin/32/1231_2.png) [@Pectojin](https://forum.duplicati.com/u/Pectojin)\ **Post date:** [February 3, 2018, 12:43am UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195/30 "2018-02-03T00:43:20Z")
Right now I’m not sure how much the semantic difference matters. There has only been 1 experimental release since the beta, which was in august. I think the beta has some serious issues that haven’t been addressed despite fixes being upstream and in the canary. I know it takes a lot of effort to maintain more frequent releases, but it’s always sad to have to explain to users that their issue on Github has already been solved but because their software is 5 months old they haven’t received the fix yet. I’d love to take a conversation on release cycle. I think more frequent releases will allow Duplicati to reach a stable release much quicker. Many fixes are so simple they could be pushed directly into a experimental/beta release after a little testing/validation from someone other than the committer. Keeping new features in Canary for a while is good, but fixes solve problems that may be urgent to users and should be released as soon as possible. I’ll happily donate time to help figuring out what goes into the releases or even build packages if that’s what it takes. I just don’t know how to help here. ---
**Author:** ![kenkendk](https://forum.duplicati.com/user_avatar/forum.duplicati.com/kenkendk/32/5305_2.png) [@kenkendk](https://forum.duplicati.com/u/kenkendk)\ **Post date:** [February 5, 2018, 11:17am UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195/31 "2018-02-05T11:17:36Z")
> [@Pectojin](#): > > I’d love to take a conversation on release cycle. I will just go ahead an admit that I am bad at pushing the releases and keeping a roadmap on track. What usually happens is that we agree on some features/bugfixes that should make it into the next release. Then new bugs emerge and new fixes, and then I delay the release to get those into the next version, and we end up not having a new release. Since the releases are signed with a certificate that is my personal one, I need to do the actual builds, and I will happily do so. But we need someone who decides which fixes go into what version, and which issues are blocking a release. Alternatively we can do it with a fixed scheme, say a beta each 3 months, but I think that is more likely to break things. > [@Pectojin](#): > > I’ll happily donate time to help figuring out what goes into the releases or even build packages if that’s what it takes. I just don’t know how to help here. I can give you commit access to the github repo. From there I think it is a matter of creating a branch (say releases/experimental\_v2.xxxx), base it on some relatively safe point, and maybe cherry pick commits into that. Once you are happy with the version, maybe we should start a topic in #Releases and people can chime in with opinions (in case there is something that _really_ should not go out). Then I will be the “build-bot” and press the button once we have a “go”. For a more planned approach, @kees-z made a great job collecting a number of issues that we should fix: [2.0 stable Milestone · GitHub](https://github.com/duplicati/duplicati/milestone/8) I think the milestone approach is the right way to plan and discuss releases, but I have been tied up (too much) on the concurrency branch, because it will also greatly improve code readability and make it simpler to introduce a number of larger changes that I have planned. It looks like I have managed to get it working, but Travis is having some [issues](https://github.com/travis-ci/travis-ci/issues/9112), so I cannot verify that it actually works. ---
**Author:** ![Pectojin](https://forum.duplicati.com/user_avatar/forum.duplicati.com/pectojin/32/1231_2.png) [@Pectojin](https://forum.duplicati.com/u/Pectojin)\ **Post date:** [February 5, 2018, 2:15pm UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195/32 "2018-02-05T14:15:33Z")
Perhaps I should start a new thread on #releases to discuss all the details 🙂 I’ll write something up after work today. ---
**Author:** ![Pectojin](https://forum.duplicati.com/user_avatar/forum.duplicati.com/pectojin/32/1231_2.png) [@Pectojin](https://forum.duplicati.com/u/Pectojin)\ **Post date:** [February 5, 2018, 6:53pm UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195/33 "2018-02-05T18:53:45Z")
I created the thread here: [Discussion: release cycle](https://forum.duplicati.com/t/discussion-release-cycle/2365) It’s a bit long, I’m hoping to get it updated once I get some feedback from the community. Does it make sense to turn it into a “wiki” so that other people can edit? ---
**Author:** ![Tekki](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/t/da6949/32.png) [@Tekki](https://forum.duplicati.com/u/Tekki)\ **Post date:** [February 11, 2018, 10:45pm UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195/34 "2018-02-11T22:45:59Z")
**Heads up!:** We decided to change the `--retention-policy` option a bit: It now deletes backups older than any timeframe by itself, so there is no need anymore to also use `--keep-time`. And to not have any confusing situations with people combining `--retention-policy` with `--keep-time` and `--keep-versions`, these are now all mutual exclusive, meaning you get an error if you have them combined in any way. ---
**Author:** ![JonMikelV](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/j/439d5e/32.png) [@JonMikelV](https://forum.duplicati.com/u/JonMikelV)\ **Post date:** [February 12, 2018, 2:09am UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195/35 "2018-02-12T02:09:58Z")
So if we want to have a “never delete” type setting we’d want to include something like “99Y:1Y” in our custom retention string? ---
**Author:** ![Tekki](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/t/da6949/32.png) [@Tekki](https://forum.duplicati.com/u/Tekki)\ **Post date:** [February 12, 2018, 8:16am UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195/36 "2018-02-12T08:16:54Z")
> [@JonMikelV](#): > > So if we want to have a “never delete” type setting we’d want to include something like “99Y:1Y” in our custom retention string? Yep, exactly. Oh and the most current backup will of course still always be kept, even if it’s outside of any timeframe (unless `--allow-full-removal` is enabled) ---
**Author:** ![JonMikelV](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/j/439d5e/32.png) [@JonMikelV](https://forum.duplicati.com/u/JonMikelV)\ **Post date:** [February 12, 2018, 4:23pm UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195/37 "2018-02-12T16:23:05Z")
> [@Tekki](#): > > the most current backup will of course still always be kept Thanks for the clarification! For those that might not follow the difference, here’s a quick summary of how `--retention-policy` cleanup handles “end of life” of a file / version: - any file still existing in the source will always have at least the most recent version in the backup even if that most recent backup is older than retention policy limits. (Unless `--allow-full-removal` is enabled, in which case backup versions older than retention rules allow WILL be removed from the backup even if the file still exists in the source.) - any file deleted from the source will subsequently be deleted from the backup when it reaches the retention policy limit. This means that if you want to keep files “forever” (like CrashPlan allows) you should make sure your retention policy includes a very long time frame. For example, adding “99Y:99Y” means a deleted (from the source) file won’t be deleted until the version in the backup is over 99 years old. The “99Y” before the colon means keep “things around for 99 years”. The “99Y” after the colon means “keep 1 version for every 99 years”. ---
**Author:** ![Pectojin](https://forum.duplicati.com/user_avatar/forum.duplicati.com/pectojin/32/1231_2.png) [@Pectojin](https://forum.duplicati.com/u/Pectojin)\ **Post date:** [February 12, 2018, 4:38pm UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195/38 "2018-02-12T16:38:25Z")
> [@JonMikelV](#): > > (Unless --allow-full-removal is enabled, in which case backup versions older than retention rules allow WILL be removed from the backup even if the file still exists in the source.) Wait, so this would mean that with a “1Y:1W” retention plan and `--allow-full-removal` enabled I would lose the ability to restore files that have not changed in the past year even they still exist on the system? I’m curious about when that use case would ever be desired. > [@JonMikelV](#): > > The “1Y” after the colon means “keep 1 version for every year". FTFY 😉 ---
**Author:** ![JonMikelV](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/j/439d5e/32.png) [@JonMikelV](https://forum.duplicati.com/u/JonMikelV)\ **Post date:** [February 12, 2018, 4:55pm UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195/39 "2018-02-12T16:55:00Z")
> [@Pectojin](#): > > with a “1Y:1W” retention plan and --allow-full-removal enabled I would lose the ability to restore files that have not changed in the past year even they still exist on the system? As I understand it, yep. I personally wouldn’t do that, but if somebody had very limited destination space or wanted to simulate a “full backup” but for some reason didn’t want to use `--keep-versions=1` I suppose it would do the trick. I supposed it could be geared more towards manual tasks (single file cleanup and the like) than every day backup use. Though even then `purge` would probably be a better choice. > [@Pectojin](#): > > FTFY Thanks! ---
**Author:** ![renestach](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/r/848f3c/32.png) [@renestach](https://forum.duplicati.com/u/renestach)\ **Post date:** [February 12, 2018, 6:19pm UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195/40 "2018-02-12T18:19:08Z")
> So if we want to have a “never delete” type setting we’d want to include something like “99Y:1Y” in our custom retention string? No, then you just select “Keep all backups” in the UI. 😛 [Previous page](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195.md?page=1) [Next page](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195.md?page=3)