Problem with Smart Retention on Linux?

Hello to All,

I’m new to Duplicati, using the latest version (2.0.3.3_beta_2018-04-02) on both Linux and Windows 7 and I am doing tests with the Smart Backup Retention (–retention-policy=“1W:1D,4W:1W,12M:1M”).

On Windows everything seems to work as expected (so far). However, on Linux every time I run a backup it seems to delete the previous one. I say “seems” because I am basing this on the list of backups shown on the “Restore files” page. Only the original backup and most recent one are shown. If I run a new backup, the most recent is replaced by the latest one. Is this known behavior?

If it helps, this is the version of mono I am running:
Mono JIT compiler version 4.6.2 (Debian 4.6.2.7+dfsg-1)

The destination for my files is B2 and I am using the built in AES-256 encryption option.

Thanks for any advice you can provide!

Hey netadmin,

how often are you running backups? With the above mentioned retention policy, the behaviour would be expected, if you run the backup multiple times within a day.

The 1W:1D means: Within the timeframe of 1 week (equalling 7 days or 10080 minutes) only keep backups that are at least 1 day (equalling 24 hours or 1440 minutes) apart from eachother.

If this happens even with backups that are multiple days apart from eachother, then see if the log files give you more information. For this set the log-level option to Profiling and log-file option to a path of your choice. This will log the decisions the Retention Policy algorithm made (among many many other things):

For example:

Information: [Retention Policy]: Start checking if backups can be removed
Information: [Retention Policy]: Time frames and intervals pairs: 14.00:00:00 / Keep all, 60.00:00:00 / 7.00:00:00, 365.00:00:00 / 31.00:00:00
Information: [Retention Policy]: Backups to consider: 04.05.2018 10:31:23, 04.05.2018 09:41:45
Profiling: [Retention Policy]: Next time frame and interval pair: 14.00:00:00 / Keep all
Profiling: [Retention Policy]: Backups in this time frame: 04.05.2018 09:41:45, 04.05.2018 10:31:23
Profiling: [Retention Policy]: Keeping backup: 04.05.2018 09:41:45
Profiling: [Retention Policy]: Keeping backup: 04.05.2018 10:31:23
Profiling: [Retention Policy]: Next time frame and interval pair: 60.00:00:00 / 7.00:00:00
Profiling: [Retention Policy]: Backups in this time frame: 
Profiling: [Retention Policy]: Next time frame and interval pair: 365.00:00:00 / 31.00:00:00
Profiling: [Retention Policy]: Backups in this time frame: 
Information: [Retention Policy]: Backups outside of all time frames and thus getting deleted: 
Information: [Retention Policy]: All backups to delete:

I ran into this myself when doing manually run test backups so I’ve started adding 1D:U to my rules so that if I run 2 or 3 test backups in a day it doesn’t through out the previous one for that day.

Just in case that’s not clear, in @netadmin’s case that would be entered as –retention-policy="1D:U,1W:1D,4W:1W,12M:1M".


Just in case somebody bumps into this later and gets warnings about the --log-level parameter, Duplicati versions AFTER 2.0.3.3 beta have updated logging code that uses the following parameters instead of --log-level:

--log-file-log-level (basically replaces --log-level)
Log file information level
Default value: “Warning”

--log-file-log-filter (allows filtering of what goes into the log file)
This option accepts filters that removes or includes messages regardless of their log level. Multiple filters are supported by separating with ;. Filters are matched against the log tag and assumed to be including, unless they start with ‘-’. Regular expressions are supported within hard braces. Example: “+Path*;+Mail;-[.*DNS]”
Default value: “”

Thanks for the replies guys. I finally found a few moments to retest this. I’m pretty sure I had waited until the next day to rerun my backup (manually), but then I realized that it uses the exact measurement of 86400 seconds to determine that a day has passed, and it’s not just a check that the next calendar day has arrived.

So, I tried JonMikelV’s suggestion and added “1D:U” to the retention policy. Unfortunately, and much to my surprise, it hasn’t made a difference.

Below is the last bit of the debug. It clearly makes the decision to delete the backup from April 30th, but why? How is that outside of my time frame?

Thanks!

EndTime: 08/05/2018 17:32:59 (1525800779)
BeginTime: 08/05/2018 16:29:46 (1525796986)
Duration: 01:03:12.9989860
Messages: [
    [Retention Policy]: Start checking if backups can be removed,
    [Retention Policy]: Time frames and intervals pairs: 1.00:00:00 / Keep all, 7.00:00:00 / 1.00:00:00, 28.00:00:00 / 7.00:00:00, 365.00:00:00 / 31.00:00:00,
    [Retention Policy]: Backups to consider: 30/04/2018 19:13:17, 26/04/2018 16:11:33,
    [Retention Policy]: Backups outside of all time frames and thus getting deleted: ,
    [Retention Policy]: All backups to delete: 30/04/2018 19:13:17,
    Deleting 1 remote fileset(s) ...,
    Deleted 1 remote fileset(s),
    Found 0 fully deletable volume(s),
    Found 1 small volumes(s) with a total size of 319.36 KB,
    Found 31 volume(s) with a total of 28.79% wasted space (5.85 GB of 20.33 GB),
    Compacting because there is 28.79% wasted space and the limit is 25%,
    Downloaded 31 file(s) with a total size of 1.43 GB, deleted 62 file(s) with a total size of 1.44 GB, and compacted to 4 file(s) with a size of 81.57 MB, which reduced storage by 58 file(s) and 1.36 GB,

I wonder how hard it would be to add what time-frame rule caused the deletion (similar to how files show what rules excluded or included them)?

At the time you run the backup (08/05/2018 17:32:59) both backups (30/04/2018 19:13:17, 26/04/2018 16:11:33) are within the 4W:1W timeframe. The 1W:1D timeframe ended on 01/05/2018 17:32:59 and none of the backups fall into that.

26/04/2018 16:11:33 is the oldest and gets set as the start point. From there the next backup is not allowed to be any older than one week (=7 days = 604,800 seconds), which would be 03/05/2018 16:11:33.

Since the backup 30/04/2018 19:13:17 is further in the past than this (only roughly 4 days), it gets deleted.

That actually already gets logged one the Profiling level (see my example a bit further above), though I’m not sure how it works with the new logging system, that was implemented after the current beta. Didn’t have any time to try the newest Canary builds :-/

Tekki

Do you mean the “rules in play” line?

[Retention Policy]: Time frames and intervals pairs: 1.00:00:00 / Keep all, 7.00:00:00 / 1.00:00:00, 28.00:00:00 / 7.00:00:00, 365.00:00:00 / 31.00:00:00,

If so, I was thinking more along the lines of this indicating that the 7.00:00:00 / 1.00:00:00 rule is what caused the 30/04/2018 19:13:17 backup to be deleted:

[Retention Policy]: All backups to delete: 30/04/2018 19:13:17 [7D:1D],