Backup Retention not working correctly

Support
1

I had the backup retention set to smart but it was only keeping one backup, I set it to custom and used 7D:1D,4W:1W,36M:1M but it is still only keeping 5 backups

2

Can you Export your backup job “as command-line” and paste it here? Mask sensitive information of course (passwords, etc).

3

Hello @rdwild and welcome to the forum!

How often were backups made? Retention doesn’t make backups, it only decides which to retain or delete.

You can see its steps at the end of any backup. Easiest way is About --> Show log --> Live --> Information.

Below are some log lines done a more long-term way using –log-file and –log-file-log-level. Information isn’t terribly verbose, but Profiling level is, so you might want to stick with Information unless you need all details.

Example:

2019-04-30 22:02:45 -04 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-StartCheck]: Start checking if backups can be removed
2019-04-30 22:02:45 -04 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-FramesAndIntervals]: Time frames and intervals pairs: 7.00:00:00 / 1.00:00:00, 28.00:00:00 / 7.00:00:00, 365.00:00:00 / 31.00:00:00
2019-04-30 22:02:45 -04 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-BackupList]: Backups to consider: 4/30/2019 9:00:00 PM, 4/30/2019 7:00:00 PM, 4/29/2019 7:00:00 PM, 4/28/2019 7:00:00 PM, 4/26/2019 3:54:41 PM, 4/15/2019 7:19:56 AM, 4/7/2019 6:16:14 PM, 3/29/2019 5:18:13 PM
2019-04-30 22:02:45 -04 - [Profiling-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-NextTimeAndFrame]: Next time frame and interval pair: 7.00:00:00 / 1.00:00:00
2019-04-30 22:02:45 -04 - [Profiling-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-BackupsInFrame]: Backups in this time frame: 4/26/2019 3:54:41 PM, 4/28/2019 7:00:00 PM, 4/29/2019 7:00:00 PM, 4/30/2019 7:00:00 PM, 4/30/2019 9:00:00 PM
2019-04-30 22:02:45 -04 - [Profiling-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-KeepBackups]: Keeping backup: 4/26/2019 3:54:41 PM
2019-04-30 22:02:45 -04 - [Profiling-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-KeepBackups]: Keeping backup: 4/28/2019 7:00:00 PM
2019-04-30 22:02:45 -04 - [Profiling-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-KeepBackups]: Keeping backup: 4/29/2019 7:00:00 PM
2019-04-30 22:02:45 -04 - [Profiling-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-KeepBackups]: Keeping backup: 4/30/2019 7:00:00 PM
2019-04-30 22:02:45 -04 - [Profiling-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-DeletingBackups]: Deleting backup: 4/30/2019 9:00:00 PM
2019-04-30 22:02:45 -04 - [Profiling-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-NextTimeAndFrame]: Next time frame and interval pair: 28.00:00:00 / 7.00:00:00
2019-04-30 22:02:45 -04 - [Profiling-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-BackupsInFrame]: Backups in this time frame: 4/7/2019 6:16:14 PM, 4/15/2019 7:19:56 AM
2019-04-30 22:02:45 -04 - [Profiling-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-KeepBackups]: Keeping backup: 4/7/2019 6:16:14 PM
2019-04-30 22:02:45 -04 - [Profiling-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-KeepBackups]: Keeping backup: 4/15/2019 7:19:56 AM
2019-04-30 22:02:45 -04 - [Profiling-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-NextTimeAndFrame]: Next time frame and interval pair: 365.00:00:00 / 31.00:00:00
2019-04-30 22:02:45 -04 - [Profiling-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-BackupsInFrame]: Backups in this time frame: 3/29/2019 5:18:13 PM
2019-04-30 22:02:45 -04 - [Profiling-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-KeepBackups]: Keeping backup: 3/29/2019 5:18:13 PM
2019-04-30 22:02:45 -04 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-BackupsToDelete]: Backups outside of all time frames and thus getting deleted: 
2019-04-30 22:02:45 -04 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-AllBackupsToDelete]: All backups to delete: 4/30/2019 9:00:00 PM
4

“C:\Program Files\Duplicati 2\Duplicati.CommandLine.exe” backup “s3s://xxxxxx-backup/?s3-server-name=s3.amazonaws.com&s3-location-constraint=us-west-1&s3-storage-class=&auth-username=xxxxx&auth-password=xxxxx” “E:\NextCloud\Personal\” “E:\NextCloud\Photos\” --backup-name=“Test Backup - xxxxx” --dbpath=“C:\Users\rod.wild.xxxx\AppData\Local\Duplicati\88798079817970746882.sqlite” --encryption-module=aes --compression-module=zip --dblock-size=500MB --passphrase=“xxxxx” --retention-policy=“7D:1D,4W:1W,36M:1M” --disable-module=console-password-input

5

Excellent point! I didn’t even think that maybe jobs weren’t scheduled lol.

6

@rdwild your job definition looks fine to me - but yeah how often are backups scheduled to run?

7

Backups are scheduled to run every day at 1:00 AM (the computer is always on). Funny thing is, its now working or at least it seems to be. I think it started working after I ran a manual backup but I will try it on another machine and report back in a few weeks when I have more details.

  • May 9, 2019 1:03 AM: Result

  • May 8, 2019 1:00 AM: Result

  • May 7, 2019 3:11 PM: Result

  • May 7, 2019 3:00 PM: Result

  • May 7, 2019 1:03 AM: Result

  • May 6, 2019 1:03 AM: Result

  • May 5, 2019 1:08 AM: Result

  • May 4, 2019 8:48 AM: Result

  • Apr 27, 2019 1:04 AM: Result

  • Apr 26, 2019 1:03 AM: Result

  • Apr 25, 2019 1:03 AM: Result

  • Apr 24, 2019 1:00 AM: Result

  • Apr 23, 2019 1:00 AM: Result

  • Apr 22, 2019 1:03 AM: Result

  • Apr 21, 2019 1:03 AM: Result

  • Apr 20, 2019 1:03 AM: Result

  • Apr 19, 2019 1:03 AM: Result

  • Apr 18, 2019 1:03 AM: Result

  • Apr 17, 2019 1:03 AM: Result

  • Apr 16, 2019 1:03 AM: Result

  • Apr 15, 2019 1:03 AM: Result

  • Apr 14, 2019 1:04 AM: Result

  • Apr 13, 2019 1:03 AM: Result

  • Apr 12, 2019 1:03 AM: Result

  • Apr 11, 2019 8:51 AM: Result

  • Apr 10, 2019 9:11 AM: Result

  • Apr 10, 2019 1:12 AM: Result

  • Apr 9, 2019 1:03 AM: Result

Here is the log file for the last backup:

DeletedFiles: 0
DeletedFolders: 0
ModifiedFiles: 0
ExaminedFiles: 6897
OpenedFiles: 1
AddedFiles: 1
SizeOfModifiedFiles: 0
SizeOfAddedFiles: 84412
SizeOfExaminedFiles: 27374276418
SizeOfOpenedFiles: 84412
NotProcessedFiles: 0
AddedFolders: 0
TooLargeFiles: 0
FilesWithError: 0
ModifiedFolders: 0
ModifiedSymlinks: 0
AddedSymlinks: 0
DeletedSymlinks: 0
PartialBackup: False
Dryrun: False
MainOperation: Backup
CompactResults:
DeletedFileCount: 0
DownloadedFileCount: 0
UploadedFileCount: 0
DeletedFileSize: 0
DownloadedFileSize: 0
UploadedFileSize: 0
Dryrun: False
MainOperation: Compact
ParsedResult: Success
Version: 2.0.4.5 (2.0.4.5_beta_2018-11-28)
EndTime: 5/9/2019 1:00:08 AM (1557385208)
BeginTime: 5/9/2019 1:00:07 AM (1557385207)
Duration: 00:00:00.7595911
Messages: [
2019-05-09 01:00:00 -06 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started,
2019-05-09 01:00:00 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: (),
2019-05-09 01:00:01 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: (128 bytes),
2019-05-09 01:00:03 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b2164a710972f4a5d9b924258271ee3c6.dblock.zip.aes (81.54 KB),
2019-05-09 01:00:04 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b2164a710972f4a5d9b924258271ee3c6.dblock.zip.aes (81.54 KB),
…
]
Warnings:
Errors:
BackendStatistics:
RemoteCalls: 8
BytesUploaded: 885175
BytesDownloaded: 525278775
FilesUploaded: 3
FilesDownloaded: 3
FilesDeleted: 0
FoldersCreated: 0
RetryAttempts: 0
UnknownFileSize: 0
UnknownFileCount: 0
KnownFileCount: 131
KnownFileSize: 26184836359
LastBackupDate: 5/9/2019 1:00:00 AM (1557385200)
BackupListCount: 7
TotalQuotaSpace: 0
FreeQuotaSpace: 0
AssignedQuotaSpace: -1
ReportedQuotaError: False
ReportedQuotaWarning: False
ParsedResult: Success
Version: 2.0.4.5 (2.0.4.5_beta_2018-11-28)
Messages: [
2019-05-09 01:00:00 -06 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started,
2019-05-09 01:00:00 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: (),
2019-05-09 01:00:01 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: (128 bytes),
2019-05-09 01:00:03 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b2164a710972f4a5d9b924258271ee3c6.dblock.zip.aes (81.54 KB),
2019-05-09 01:00:04 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b2164a710972f4a5d9b924258271ee3c6.dblock.zip.aes (81.54 KB),
…
]
Warnings:
Errors:
DeleteResults:
DeletedSets:
Dryrun: False
MainOperation: Delete
ParsedResult: Success
Version: 2.0.4.5 (2.0.4.5_beta_2018-11-28)
EndTime: 5/9/2019 1:00:08 AM (1557385208)
BeginTime: 5/9/2019 1:00:06 AM (1557385206)
Duration: 00:00:01.7730210
Messages: [
2019-05-09 01:00:00 -06 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started,
2019-05-09 01:00:00 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: (),
2019-05-09 01:00:01 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: (128 bytes),
2019-05-09 01:00:03 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b2164a710972f4a5d9b924258271ee3c6.dblock.zip.aes (81.54 KB),
2019-05-09 01:00:04 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b2164a710972f4a5d9b924258271ee3c6.dblock.zip.aes (81.54 KB),
…
]
Warnings:
Errors:
RepairResults: null
TestResults:
MainOperation: Test
Verifications: [
Key: duplicati-20190509T070000Z.dlist.zip.aes
Value: ,
Key: duplicati-i1a602249084c43ec84b59453f1edd633.dindex.zip.aes
Value: ,
Key: duplicati-bc1c4d52e7daa48fa9c6242b4a2783bc6.dblock.zip.aes
Value:
]
ParsedResult: Success
Version: 2.0.4.5 (2.0.4.5_beta_2018-11-28)
EndTime: 5/9/2019 1:03:36 AM (1557385416)
BeginTime: 5/9/2019 1:00:08 AM (1557385208)
Duration: 00:03:27.6758673
Messages: [
2019-05-09 01:00:00 -06 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started,
2019-05-09 01:00:00 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: (),
2019-05-09 01:00:01 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: (128 bytes),
2019-05-09 01:00:03 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b2164a710972f4a5d9b924258271ee3c6.dblock.zip.aes (81.54 KB),
2019-05-09 01:00:04 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b2164a710972f4a5d9b924258271ee3c6.dblock.zip.aes (81.54 KB),
…
]
Warnings:
Errors:
ParsedResult: Success
Version: 2.0.4.5 (2.0.4.5_beta_2018-11-28)
EndTime: 5/9/2019 1:03:36 AM (1557385416)
BeginTime: 5/9/2019 1:00:00 AM (1557385200)
Duration: 00:03:36.4052984
Messages: [
2019-05-09 01:00:00 -06 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started,
2019-05-09 01:00:00 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: (),
2019-05-09 01:00:01 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: (128 bytes),
2019-05-09 01:00:03 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b2164a710972f4a5d9b924258271ee3c6.dblock.zip.aes (81.54 KB),
2019-05-09 01:00:04 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b2164a710972f4a5d9b924258271ee3c6.dblock.zip.aes (81.54 KB),
…
]
Warnings:
Errors:

8

Hello! Does Duplicati support 180m:1m,24h:1h,7D:1D,4W:1W? This function does not work. The minimum period is the “day” or function not correctly written? How to record “hours”, “minutes”?

9

Should work unless your’re still on 2.0.5.1_beta_2020-01-18 which had a bug and rejected m for minutes.
From About → Changelog

Fixed allowing minutes in retention rules, thanks @Pectojin

If you’re staying on 2.0.5.1 for some reason, you can rewrite 180m as 3h, and 1m as U to mean unlimited.

If it’s not above, what happens?

10

I use 2.0.6.3_beta_2021-06-17.

I think it works not correctly. It works strange … In the screenshot, the recovery points that are created for frequently changing file by the formula “180m:1m,24h:1h,7D:1D,4W:1W.”
Copies are created more often than three hours (180m: 1m), but not every day. The computer turns on every day, sometimes for a whole day, sometimes for a couple of hours.

Clipboard022
Clipboard022636Ă—539 75 KB

Where can I read more? Here, for example, there is no information about the “hours” and “minutes”.

11

Retention refers to retaining certain backup versions (and deleting others), not creating versions, which is Schedule or manual. Is there anything on the list that violates the minimum age-based interval of 1 minute going back 180m, or any of the others? I can see how it looks like 3h - 24h range is thinned to 1h spacing.

New retention policy deletes old backups in a smart way is the announcement, which might be a bit dated.

Command line help:

  --retention-policy (String): Reduce number of versions by deleting old
    intermediate backups
    Use this option to reduce the number of versions that are kept with
    increasing version age by deleting most of the old backups. The expected
    format is a comma separated list of colon separated time frame and
    interval pairs. For example the value "7D:0s,3M:1D,10Y:2M" means "For 7
    day keep all backups, for 3 months keep one backup every day, for 10
    years one backup every 2nd month and delete every backup older than
    this.". This option also supports using the specifier "U" to indicate an
    unlimited time interval.

GUI help:

image

GUI smart and custom backup retention aren’t covered #83 is a request to revise the user manual section.

EDIT:

You can see the retention policy being applied at end of backup in an Information-level log-file. For example:

2021-10-06 09:45:56 -04 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-StartCheck]: Start checking if backups can be removed
2021-10-06 09:45:56 -04 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-FramesAndIntervals]: Time frames and intervals pairs: 1.00:00:00 / Keep all, 7.00:00:00 / 1.00:00:00, 28.00:00:00 / 7.00:00:00, 365.00:00:00 / 31.00:00:00
2021-10-06 09:45:56 -04 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-BackupList]: Backups to consider: 10/6/2021 8:40:04 AM, 10/6/2021 7:40:16 AM, 10/6/2021 6:57:12 AM, 10/5/2021 9:40:00 PM, 10/5/2021 8:40:04 PM, 10/5/2021 7:40:27 PM, 10/5/2021 6:40:07 PM, 10/5/2021 5:40:06 PM, 10/5/2021 4:40:14 PM, 10/5/2021 3:40:03 PM, 10/5/2021 2:40:05 PM, 10/5/2021 1:40:09 PM, 10/5/2021 12:40:20 PM, 10/5/2021 11:40:02 AM, 10/5/2021 10:40:00 AM, 10/5/2021 9:40:13 AM, 10/4/2021 10:40:03 AM, 10/3/2021 10:40:00 AM, 10/2/2021 10:40:00 AM, 10/1/2021 9:40:02 AM, 9/30/2021 9:40:02 AM, 9/29/2021 9:40:00 AM, 9/27/2021 8:40:00 AM, 9/26/2021 9:22:14 PM, 9/19/2021 8:40:00 PM, 9/12/2021 5:40:02 PM, 9/8/2021 9:41:05 AM, 8/7/2021 8:40:00 AM, 8/5/2021 1:40:04 PM, 7/4/2021 6:33:36 PM, 7/1/2021 9:40:03 AM, 6/24/2021 7:40:55 PM, 6/20/2021 9:33:01 AM, 6/20/2021 8:02:19 AM, 6/20/2021 6:58:27 AM, 5/25/2021 11:40:02 AM, 5/22/2021 1:58:23 PM, 5/22/2021 1:16:35 PM, 4/19/2021 12:40:01 PM, 4/2/2021 9:40:02 PM, 3/14/2021 2:40:01 PM, 2/6/2021 10:40:04 AM, 1/1/2021 10:40:04 AM, 11/26/2020 7:44:53 AM, 10/17/2020 8:40:00 PM, 10/14/2020 6:40:02 AM
2021-10-06 09:45:56 -04 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-BackupsToDelete]: Backups outside of all time frames and thus getting deleted:
2021-10-06 09:45:56 -04 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-AllBackupsToDelete]: All backups to delete: 10/5/2021 9:40:13 AM, 9/29/2021 9:40:00 AM
2021-10-06 09:45:56 -04 - [Information-Duplicati.Library.Main.Operation.DeleteHandler-DeleteRemoteFileset]: Deleting 2 remote fileset(s) …

For a quicker peek (in reverse-chronological order), you can see About → Show log → Live → Information.

1 Like
12

Thanks! Maybe I understanded. I will test.

No. I mistake.