# New retention policy deletes old backups in a smart way **URL:** \ **Category:** Announcements\ **Created:** [January 23, 2018, 6:47pm UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195 "2018-01-23T18:47:03Z")\ **Posts on this page:** 20\ **Page:** 3
**Author:** ![Pectojin](https://forum.duplicati.com/user_avatar/forum.duplicati.com/pectojin/32/1231_2.png) [@Pectojin](https://forum.duplicati.com/u/Pectojin)\ **Post date:** [February 12, 2018, 6:23pm UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195/41 "2018-02-12T18:23:49Z")
Well, you can want retention thinning and unlimited age šŸ™‚ Although at 99 years I think itā€™s close to pointless. By the way, does anyone know if 99 is max? ---
**Author:** ![JonMikelV](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/j/439d5e/32.png) [@JonMikelV](https://forum.duplicati.com/u/JonMikelV)\ **Post date:** [February 12, 2018, 6:30pm UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195/42 "2018-02-12T18:30:38Z")
> [@Pectojin](#): > > Well, you can want retention thinning and unlimited age Exactly. I want to be able to go back to previous versions, but after a few weeks I donā€™t need EVERY version. On the flip side I have family members who delete stuff they only use every few years (think 5 year reunions), so donā€™t realize itā€™s missing for quite some time. The retention policy flexibility lets me handle both scenarios with a single backup config. Yay! ---
**Author:** ![renestach](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/r/848f3c/32.png) [@renestach](https://forum.duplicati.com/u/renestach)\ **Post date:** [February 12, 2018, 6:32pm UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195/43 "2018-02-12T18:32:52Z")
Would it make sense to introduce a specific keyword like 1W:1D,ELSE:1W ? ---
**Author:** ![drakar2007](https://forum.duplicati.com/user_avatar/forum.duplicati.com/drakar2007/32/230_2.png) [@drakar2007](https://forum.duplicati.com/u/drakar2007)\ **Post date:** [February 12, 2018, 6:38pm UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195/44 "2018-02-12T18:38:07Z")
> [@JonMikelV](#): > > The retention policy flexibility lets me handle both scenarios with a single backup config. Yay! Can anyone confirm whether the new retention policy functionality now handles the case of accidentally-deleted single files? I believe the first version would _not_ retain the last backup of an individual file post deletion if it wasnā€™t in the ā€œrightā€ backup set - is it any different now? ---
**Author:** ![Pectojin](https://forum.duplicati.com/user_avatar/forum.duplicati.com/pectojin/32/1231_2.png) [@Pectojin](https://forum.duplicati.com/u/Pectojin)\ **Post date:** [February 12, 2018, 6:38pm UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195/45 "2018-02-12T18:38:14Z")
How about 1W:1D,U:1W? With U being unlimited? ---
**Author:** ![drakar2007](https://forum.duplicati.com/user_avatar/forum.duplicati.com/drakar2007/32/230_2.png) [@drakar2007](https://forum.duplicati.com/u/drakar2007)\ **Post date:** [February 12, 2018, 6:39pm UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195/46 "2018-02-12T18:39:04Z")
> [@Pectojin](#): > > How about 1W:1D,U:1W? With U being unlimited? Or just 1W:1D,200Y:1Wā€¦ ---
**Author:** ![Pectojin](https://forum.duplicati.com/user_avatar/forum.duplicati.com/pectojin/32/1231_2.png) [@Pectojin](https://forum.duplicati.com/u/Pectojin)\ **Post date:** [February 12, 2018, 6:51pm UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195/47 "2018-02-12T18:51:08Z")
I know itā€™s practically the same, but I think itā€™s still a valid point to have a _semantically_ logical expression. If you just want to keep them forever itā€™s weird that you have to make up some arbitrary number. I mean, I literally donā€™t care and the expression still made me pick an arbitrary number, that number being 10 years. And more paranoid people, like @JonMikelV, will tell you they picked 100 - but secretly they picked 9999 just to be safe šŸ˜‰ ---
**Author:** ![kees-z](https://forum.duplicati.com/user_avatar/forum.duplicati.com/kees-z/32/29_2.png) [@kees-z](https://forum.duplicati.com/u/kees-z)\ **Post date:** [February 12, 2018, 6:52pm UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195/48 "2018-02-12T18:52:20Z")
> [@JonMikelV](#): > > > [@Pectojin](#): > > > > with a ā€œ1Y:1Wā€ retention plan and --allow-full-removal enabled I would lose the ability to restore files that have not changed in the past year even they still exist on the system? > > As I understand it, yep. Thatā€™s not how it works, at least, I hope so. Backup retention is not about _file versions_, it thins out _backup versions_ (or restore points, whatever you want to name it). Every backup version is a representation of everything included in your source selection. If you delete a file and start a backup job, that file will not be part of the most recent backup, but it is still part of all other backup versions (provided that the oldest backup is more recent than the file creation date). When a file exists on your system, it will be part of the most recent backup, You can restore it from every backup version of the last year. Only backup versions that are older than a year are deleted. So if you _modified_ a file more than a year ago and didnā€™t modify it since than, only the current version is in your backup. If you accidentally deleted or corrupted thast file, you can successfully restore it from any of the backup versions. ---
**Author:** ![kees-z](https://forum.duplicati.com/user_avatar/forum.duplicati.com/kees-z/32/29_2.png) [@kees-z](https://forum.duplicati.com/u/kees-z)\ **Post date:** [February 12, 2018, 7:11pm UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195/49 "2018-02-12T19:11:38Z")
> [@Pectojin](#): > > How about 1W:1D,U:1W? With U being unlimited? That would be great! The `U` value could also be used to replace `--keep-versions`. - `--keep-versions=30` could be specified as `--retention-policy=U:30`. `--retention-policy=1Y:1W;U:1M:30` would mean: keep 1 weekly backup for the last year, from then 1 monthly backup, but never delete backups if there are less than 30. Discussed at [Github](https://github.com/duplicati/duplicati/issues/3008): > Somebody has his backup job set to run every day and has set --retention-policy=1M:1D. Usually that would mean 30 backups are kept, if the PC is turned on all the time and created a new backup daily. > Now that person goes on holidays for a month. Upon return Duplicati will then delete all backups except the newest one, since they are all older than a month. Not seen as a problem in that discussion, but I guess for most people itā€™s unintended behaviour in this scenario. Adding a minimum number of backup versions would resolve that. ---
**Author:** ![Pectojin](https://forum.duplicati.com/user_avatar/forum.duplicati.com/pectojin/32/1231_2.png) [@Pectojin](https://forum.duplicati.com/u/Pectojin)\ **Post date:** [February 12, 2018, 7:26pm UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195/50 "2018-02-12T19:26:22Z")
> [@kees-z](#): > > ā€“keep-versions=30 could be specified as --retention-policy=U:30. I hadnā€™t thought about U:number, only U:date. I guess it reads as ā€œunlimited keep 30ā€, but I initially misread it as ā€œkeep one backup for every 30 days foreverā€ because I missed that there was no letter. Iā€™m not sure if itā€™s more user friendly or if it would lead to confusion. I think in that Github thread we ended up deciding that `--keep-versions` should not become legacy exactly because itā€™s so much more easy to understand than custom repressions. Also would `1M:30,1Y:1M` be valid if `1M:30` is valid? It then reads ā€œKeep no more than 30 versions (deleting the oldest) for 1 month, then keep 1 per month for 1 yearā€. If one is valid and the other isnā€™t, then I think itā€™s confusing because itā€™s inconsistent. At the same time it seems weird if you make 30 backups in a row one day, then there is no backups left from the previous 29 days of the month that can be turned into `1Y:1M`. In the case of U for example Iā€™d say `1M:U,1Y:1M` is valid because that then replaces `1M:0s,1Y:1M` - Which by the way I always thought was poor semantic form to say ā€œunlimitedā€ by saying `0s` and I think it even works with `0D`, `0M`, or `0Y`, which all looks pretty confusing to me. > [@kees-z](#): > > 1Y:1W;U:1M:30 I think this might be a complicated way of writing `1Y:1W,30M:1M. Although I guess **technically** , if you missed a full month of backups, then youā€™d only have 29 backups after two and a half year of backups, instead of 30 (then going back 31 months). But how good a use case is that? Would it make sense? ---
**Author:** ![kees-z](https://forum.duplicati.com/user_avatar/forum.duplicati.com/kees-z/32/29_2.png) [@kees-z](https://forum.duplicati.com/u/kees-z)\ **Post date:** [February 12, 2018, 8:19pm UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195/51 "2018-02-12T20:19:20Z")
My main thoughts/concerns are: - Specifying a number should Always main: I want _at least_ this number of versions and not: I want _no more_ than this number of versions. So `1M:30,1Y:1M` should not mean ā€œDelete all versions older than the first 30 versionsā€, but ā€œStop deleting when there are 30 remaining versionsā€. This will prevent deleting more versions than intended. - I was thinking about some way to define the _total_ number of versions, not the versions in a specific time frame. Not thought of the exact syntax, but somehow a trailing number could be supplied, indicating the minimum number of versions to keep. You could add an optional `K:` to supply the minimum total number of versions. Alternatively, you could add an optional third value for each time frame, so `1M:1W` would mean "one backup for every week last month, and `1M:1W:3` would be interpreted as ā€œone backup for every week last month, but stop deleting backups this months if there are 3 remaining backups in this time frameā€. My main concern is a common use case like `1M:1D`. If I choose this policy, my intention is that I have (around) 30 versions to choose from when doing a restore operation. If my computer is switched of for 1 month (holiday), all backup versions are deleted except 1, resulting in destroying all file versions with doing _nothing_. I see this as a potential problem. Combining with `--keep-versions` is not supported, so there is no way to keep my retention policy (if I backup twice a day, 50% can be deleted) **and** keep at least 20 versions. Something like `1M:1D;K:20` would resolve that. ---
**Author:** ![Pectojin](https://forum.duplicati.com/user_avatar/forum.duplicati.com/pectojin/32/1231_2.png) [@Pectojin](https://forum.duplicati.com/u/Pectojin)\ **Post date:** [February 12, 2018, 8:25pm UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195/52 "2018-02-12T20:25:00Z")
> [@kees-z](#): > > ā€œStop deleting when there are 30 remaining versionsā€ Aaaaah, now that makes sense! I didnā€™t get that part from the example, but looking back I see what you meant now šŸ™‚ Then something like `1M:30` make sense and it also answers my concerns about it working in any order. Although it cannot be `30:1M`, that makes no sense, right? > [@kees-z](#): > > You could add an optional K: to supply the minimum total number of versions. I think a letter definition will be good. `1M:K30` cannot possibly be a typo where `1M:30D` was accidentally turned into `1M:30` by deleting the letter. It all makes a lot of sense to me now, though. Unlike `U`, which is semantical, `K` actually supports purposes that were possible by combining retention with keep policy. Except this is more flexible than retention+keep policy. ---
**Author:** ![kees-z](https://forum.duplicati.com/user_avatar/forum.duplicati.com/kees-z/32/29_2.png) [@kees-z](https://forum.duplicati.com/u/kees-z)\ **Post date:** [February 12, 2018, 8:36pm UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195/53 "2018-02-12T20:36:27Z")
Thatā€™s exactly what I tried to point out. Sorry if I was unclear about that, but this is quite complex stuff. A small change can have all kinds of unwanted side effects. Didnā€™t think too much about how to resolve it in detail, but Iā€™m concerned a bit about the number of versions that could be deleted unintentionally. I guess the syntax can be improved. For example: when choosing a letter for versions to keep, you could choose K (keep), V (versions), N (number of versions) and so on. Itā€™s just a thought. ---
**Author:** ![Pectojin](https://forum.duplicati.com/user_avatar/forum.duplicati.com/pectojin/32/1231_2.png) [@Pectojin](https://forum.duplicati.com/u/Pectojin)\ **Post date:** [February 12, 2018, 8:58pm UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195/54 "2018-02-12T20:58:09Z")
> [@kees-z](#): > > this is quite complex stuff. It is. Itā€™s the classical problem when writing software for arbitrary inputs. There are millions of potential inputs and they all have to work consistently, but they should also cover every type of use case. Itā€™s good to be able to discuss it šŸ™‚ > [@kees-z](#): > > Didnā€™t think too much about how to resolve it in detail, but Iā€™m concerned a bit about the number of versions that could be deleted unintentionally. I think thatā€™s a good point. > [@kees-z](#): > > I guess the syntax can be improved. For example: when choosing a letter for versions to keep, you could choose K (keep), V (versions), N (number of versions) and so on. Itā€™s just a thought. I think itā€™s best to settle on one. The syntax is already a bit overwhelming and Iā€™ll admit I spent a good 5-10 minutes designing my first retention policy before entirely understanding what it would mean for my backup. ---
**Author:** ![Athep](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/a/2bfe46/32.png) [@Athep](https://forum.duplicati.com/u/Athep)\ **Post date:** [February 13, 2018, 7:02pm UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195/55 "2018-02-13T19:02:13Z")
Running 2.0.2.19\_canary\_2018-02-12 In order to get the retention policy runs listed in the internal log and via the e-mail reports, i must enable this in the settings: `--log-level=Information` I this the way it is supposed to be ? ---
**Author:** ![JonMikelV](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/j/439d5e/32.png) [@JonMikelV](https://forum.duplicati.com/u/JonMikelV)\ **Post date:** [February 13, 2018, 8:49pm UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195/56 "2018-02-13T20:49:59Z")
What do you mean by ā€œretention policy runsā€? Is this any run using the `--retention-policy` parameter or runs that only do cleanup (because no file changes are needing backup)? ---
**Author:** ![Athep](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/a/2bfe46/32.png) [@Athep](https://forum.duplicati.com/u/Athep)\ **Post date:** [February 13, 2018, 9:57pm UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195/57 "2018-02-13T21:57:44Z")
With ā€œretention policy runsā€, I mean when the configured retention policy is executed - after each backup. Example excerpt from the log: `Messages: [` `[Retention Policy]: Start checking if backups can be removed,` `[Retention Policy]: Time frames and intervals pairs: ..` `[Retention Policy]: Backups to consider: 2018-02-12..` But if `--log-level=Information` not is configured, there will be no report in the log. No information at all about the retention policy result. ---
**Author:** ![JonMikelV](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/j/439d5e/32.png) [@JonMikelV](https://forum.duplicati.com/u/JonMikelV)\ **Post date:** [February 13, 2018, 10:07pm UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195/58 "2018-02-13T22:07:22Z")
Great clarification, thanks! I suspect this is by design due to the potential for a LOT of messages to come out of retention processing, but I should probably step back and let the actual developer of this functionality answer for sure. šŸ™‚ ---
**Author:** ![drakar2007](https://forum.duplicati.com/user_avatar/forum.duplicati.com/drakar2007/32/230_2.png) [@drakar2007](https://forum.duplicati.com/u/drakar2007)\ **Post date:** [February 13, 2018, 10:16pm UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195/59 "2018-02-13T22:16:02Z")
> [@Athep](#): > > But if --log-level=Information not is configured, there will be no report in the log. No information at all about the retention policy result. I find that the ā€œresultā€ log for the individual backup job itself does reflect retention policy settings - can you check that also? 1. [Click Backup Job on Home screen] 2. ā€œReportingā€ 3. ā€œShow Logā€ 4. Select any entry ending with ā€œResultā€ 5. ā€œMessagesā€ section near the bottom of the log output ---
**Author:** ![Athep](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/a/2bfe46/32.png) [@Athep](https://forum.duplicati.com/u/Athep)\ **Post date:** [February 13, 2018, 10:37pm UTC](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195/60 "2018-02-13T22:37:21Z")
This is exactly what Iā€™m talking about. Here is more excerpt from the ā€œresultā€ log: ā€¦ ā€¦ Messages: [ [Retention Policy]: Start checking if backups can be removed, [Retention Policy]: Time frames and intervals pairs: 7.00:00:00 / 00:00:00, 31.00:00:00 / 1.00:00:00, 181.00:00:00 / 7.00:00:00, 3652.00:00:00 / 31.00:00:00, [Retention Policy]: Backups to consider: 2018-02-12 00:24:23, 2018-02-11 00:06:49, 2018-02-06 23:20:03, 2018-02-01 22:49:28, 2018-01-29 23:11:58, 2018-01-26 23:39:25, 2018-01-24 23:41:15, 2018-01-23 22:08:36, 2018-01-21 23:14:47, 2017-12-27 02:08:44, 2017-12-13 00:54:57, 2017-12-03 23:25:30, 2017-11-23 01:12:00, 2017-11-15 21:59:49, 2017-11-06 00:35:11, 2017-10-29 23:48:54, 2017-10-17 23:44:29, 2017-10-09 00:40:41, 2017-09-10 23:37:07, 2017-07-25 00:14:59, 2017-06-13 23:44:50, 2017-05-11 00:32:47, 2017-03-29 22:49:06, [Retention Policy]: Backups outside of all time frames and thus getting deleted: , [Retention Policy]: All backups to delete: , ā€¦ ] Warnings: [] Errors: [] * * * Again, no information about retention policy checking and result is included if `--log-level=Information` not is configured, [Previous page](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195.md?page=2) [Next page](https://forum.duplicati.com/t/new-retention-policy-deletes-old-backups-in-a-smart-way/2195.md?page=4)