Backup Retention not working correctly

#1

I had the backup retention set to smart but it was only keeping one backup, I set it to custom and used 7D:1D,4W:1W,36M:1M but it is still only keeping 5 backups

#2

Can you Export your backup job “as command-line” and paste it here? Mask sensitive information of course (passwords, etc).

#3

Hello @rdwild and welcome to the forum!

How often were backups made? Retention doesn’t make backups, it only decides which to retain or delete.

You can see its steps at the end of any backup. Easiest way is About --> Show log --> Live --> Information.

Below are some log lines done a more long-term way using –log-file and –log-file-log-level. Information isn’t terribly verbose, but Profiling level is, so you might want to stick with Information unless you need all details.

Example:

2019-04-30 22:02:45 -04 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-StartCheck]: Start checking if backups can be removed
2019-04-30 22:02:45 -04 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-FramesAndIntervals]: Time frames and intervals pairs: 7.00:00:00 / 1.00:00:00, 28.00:00:00 / 7.00:00:00, 365.00:00:00 / 31.00:00:00
2019-04-30 22:02:45 -04 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-BackupList]: Backups to consider: 4/30/2019 9:00:00 PM, 4/30/2019 7:00:00 PM, 4/29/2019 7:00:00 PM, 4/28/2019 7:00:00 PM, 4/26/2019 3:54:41 PM, 4/15/2019 7:19:56 AM, 4/7/2019 6:16:14 PM, 3/29/2019 5:18:13 PM
2019-04-30 22:02:45 -04 - [Profiling-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-NextTimeAndFrame]: Next time frame and interval pair: 7.00:00:00 / 1.00:00:00
2019-04-30 22:02:45 -04 - [Profiling-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-BackupsInFrame]: Backups in this time frame: 4/26/2019 3:54:41 PM, 4/28/2019 7:00:00 PM, 4/29/2019 7:00:00 PM, 4/30/2019 7:00:00 PM, 4/30/2019 9:00:00 PM
2019-04-30 22:02:45 -04 - [Profiling-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-KeepBackups]: Keeping backup: 4/26/2019 3:54:41 PM
2019-04-30 22:02:45 -04 - [Profiling-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-KeepBackups]: Keeping backup: 4/28/2019 7:00:00 PM
2019-04-30 22:02:45 -04 - [Profiling-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-KeepBackups]: Keeping backup: 4/29/2019 7:00:00 PM
2019-04-30 22:02:45 -04 - [Profiling-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-KeepBackups]: Keeping backup: 4/30/2019 7:00:00 PM
2019-04-30 22:02:45 -04 - [Profiling-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-DeletingBackups]: Deleting backup: 4/30/2019 9:00:00 PM
2019-04-30 22:02:45 -04 - [Profiling-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-NextTimeAndFrame]: Next time frame and interval pair: 28.00:00:00 / 7.00:00:00
2019-04-30 22:02:45 -04 - [Profiling-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-BackupsInFrame]: Backups in this time frame: 4/7/2019 6:16:14 PM, 4/15/2019 7:19:56 AM
2019-04-30 22:02:45 -04 - [Profiling-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-KeepBackups]: Keeping backup: 4/7/2019 6:16:14 PM
2019-04-30 22:02:45 -04 - [Profiling-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-KeepBackups]: Keeping backup: 4/15/2019 7:19:56 AM
2019-04-30 22:02:45 -04 - [Profiling-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-NextTimeAndFrame]: Next time frame and interval pair: 365.00:00:00 / 31.00:00:00
2019-04-30 22:02:45 -04 - [Profiling-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-BackupsInFrame]: Backups in this time frame: 3/29/2019 5:18:13 PM
2019-04-30 22:02:45 -04 - [Profiling-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-KeepBackups]: Keeping backup: 3/29/2019 5:18:13 PM
2019-04-30 22:02:45 -04 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-BackupsToDelete]: Backups outside of all time frames and thus getting deleted: 
2019-04-30 22:02:45 -04 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-AllBackupsToDelete]: All backups to delete: 4/30/2019 9:00:00 PM
#4

“C:\Program Files\Duplicati 2\Duplicati.CommandLine.exe” backup “s3s://xxxxxx-backup/?s3-server-name=s3.amazonaws.com&s3-location-constraint=us-west-1&s3-storage-class=&auth-username=xxxxx&auth-password=xxxxx” “E:\NextCloud\Personal\” “E:\NextCloud\Photos\” --backup-name=“Test Backup - xxxxx” --dbpath=“C:\Users\rod.wild.xxxx\AppData\Local\Duplicati\88798079817970746882.sqlite” --encryption-module=aes --compression-module=zip --dblock-size=500MB --passphrase=“xxxxx” --retention-policy=“7D:1D,4W:1W,36M:1M” --disable-module=console-password-input

#5

Excellent point! I didn’t even think that maybe jobs weren’t scheduled lol.

#6

@rdwild your job definition looks fine to me - but yeah how often are backups scheduled to run?

#7

Backups are scheduled to run every day at 1:00 AM (the computer is always on). Funny thing is, its now working or at least it seems to be. I think it started working after I ran a manual backup but I will try it on another machine and report back in a few weeks when I have more details.

  • May 9, 2019 1:03 AM: Result

  • May 8, 2019 1:00 AM: Result

  • May 7, 2019 3:11 PM: Result

  • May 7, 2019 3:00 PM: Result

  • May 7, 2019 1:03 AM: Result

  • May 6, 2019 1:03 AM: Result

  • May 5, 2019 1:08 AM: Result

  • May 4, 2019 8:48 AM: Result

  • Apr 27, 2019 1:04 AM: Result

  • Apr 26, 2019 1:03 AM: Result

  • Apr 25, 2019 1:03 AM: Result

  • Apr 24, 2019 1:00 AM: Result

  • Apr 23, 2019 1:00 AM: Result

  • Apr 22, 2019 1:03 AM: Result

  • Apr 21, 2019 1:03 AM: Result

  • Apr 20, 2019 1:03 AM: Result

  • Apr 19, 2019 1:03 AM: Result

  • Apr 18, 2019 1:03 AM: Result

  • Apr 17, 2019 1:03 AM: Result

  • Apr 16, 2019 1:03 AM: Result

  • Apr 15, 2019 1:03 AM: Result

  • Apr 14, 2019 1:04 AM: Result

  • Apr 13, 2019 1:03 AM: Result

  • Apr 12, 2019 1:03 AM: Result

  • Apr 11, 2019 8:51 AM: Result

  • Apr 10, 2019 9:11 AM: Result

  • Apr 10, 2019 1:12 AM: Result

  • Apr 9, 2019 1:03 AM: Result

Here is the log file for the last backup:

DeletedFiles: 0
DeletedFolders: 0
ModifiedFiles: 0
ExaminedFiles: 6897
OpenedFiles: 1
AddedFiles: 1
SizeOfModifiedFiles: 0
SizeOfAddedFiles: 84412
SizeOfExaminedFiles: 27374276418
SizeOfOpenedFiles: 84412
NotProcessedFiles: 0
AddedFolders: 0
TooLargeFiles: 0
FilesWithError: 0
ModifiedFolders: 0
ModifiedSymlinks: 0
AddedSymlinks: 0
DeletedSymlinks: 0
PartialBackup: False
Dryrun: False
MainOperation: Backup
CompactResults:
DeletedFileCount: 0
DownloadedFileCount: 0
UploadedFileCount: 0
DeletedFileSize: 0
DownloadedFileSize: 0
UploadedFileSize: 0
Dryrun: False
MainOperation: Compact
ParsedResult: Success
Version: 2.0.4.5 (2.0.4.5_beta_2018-11-28)
EndTime: 5/9/2019 1:00:08 AM (1557385208)
BeginTime: 5/9/2019 1:00:07 AM (1557385207)
Duration: 00:00:00.7595911
Messages: [
2019-05-09 01:00:00 -06 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started,
2019-05-09 01:00:00 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: (),
2019-05-09 01:00:01 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: (128 bytes),
2019-05-09 01:00:03 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b2164a710972f4a5d9b924258271ee3c6.dblock.zip.aes (81.54 KB),
2019-05-09 01:00:04 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b2164a710972f4a5d9b924258271ee3c6.dblock.zip.aes (81.54 KB),

]
Warnings:
Errors:
BackendStatistics:
RemoteCalls: 8
BytesUploaded: 885175
BytesDownloaded: 525278775
FilesUploaded: 3
FilesDownloaded: 3
FilesDeleted: 0
FoldersCreated: 0
RetryAttempts: 0
UnknownFileSize: 0
UnknownFileCount: 0
KnownFileCount: 131
KnownFileSize: 26184836359
LastBackupDate: 5/9/2019 1:00:00 AM (1557385200)
BackupListCount: 7
TotalQuotaSpace: 0
FreeQuotaSpace: 0
AssignedQuotaSpace: -1
ReportedQuotaError: False
ReportedQuotaWarning: False
ParsedResult: Success
Version: 2.0.4.5 (2.0.4.5_beta_2018-11-28)
Messages: [
2019-05-09 01:00:00 -06 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started,
2019-05-09 01:00:00 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: (),
2019-05-09 01:00:01 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: (128 bytes),
2019-05-09 01:00:03 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b2164a710972f4a5d9b924258271ee3c6.dblock.zip.aes (81.54 KB),
2019-05-09 01:00:04 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b2164a710972f4a5d9b924258271ee3c6.dblock.zip.aes (81.54 KB),

]
Warnings:
Errors:
DeleteResults:
DeletedSets:
Dryrun: False
MainOperation: Delete
ParsedResult: Success
Version: 2.0.4.5 (2.0.4.5_beta_2018-11-28)
EndTime: 5/9/2019 1:00:08 AM (1557385208)
BeginTime: 5/9/2019 1:00:06 AM (1557385206)
Duration: 00:00:01.7730210
Messages: [
2019-05-09 01:00:00 -06 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started,
2019-05-09 01:00:00 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: (),
2019-05-09 01:00:01 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: (128 bytes),
2019-05-09 01:00:03 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b2164a710972f4a5d9b924258271ee3c6.dblock.zip.aes (81.54 KB),
2019-05-09 01:00:04 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b2164a710972f4a5d9b924258271ee3c6.dblock.zip.aes (81.54 KB),

]
Warnings:
Errors:
RepairResults: null
TestResults:
MainOperation: Test
Verifications: [
Key: duplicati-20190509T070000Z.dlist.zip.aes
Value: ,
Key: duplicati-i1a602249084c43ec84b59453f1edd633.dindex.zip.aes
Value: ,
Key: duplicati-bc1c4d52e7daa48fa9c6242b4a2783bc6.dblock.zip.aes
Value:
]
ParsedResult: Success
Version: 2.0.4.5 (2.0.4.5_beta_2018-11-28)
EndTime: 5/9/2019 1:03:36 AM (1557385416)
BeginTime: 5/9/2019 1:00:08 AM (1557385208)
Duration: 00:03:27.6758673
Messages: [
2019-05-09 01:00:00 -06 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started,
2019-05-09 01:00:00 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: (),
2019-05-09 01:00:01 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: (128 bytes),
2019-05-09 01:00:03 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b2164a710972f4a5d9b924258271ee3c6.dblock.zip.aes (81.54 KB),
2019-05-09 01:00:04 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b2164a710972f4a5d9b924258271ee3c6.dblock.zip.aes (81.54 KB),

]
Warnings:
Errors:
ParsedResult: Success
Version: 2.0.4.5 (2.0.4.5_beta_2018-11-28)
EndTime: 5/9/2019 1:03:36 AM (1557385416)
BeginTime: 5/9/2019 1:00:00 AM (1557385200)
Duration: 00:03:36.4052984
Messages: [
2019-05-09 01:00:00 -06 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started,
2019-05-09 01:00:00 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: (),
2019-05-09 01:00:01 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: (128 bytes),
2019-05-09 01:00:03 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b2164a710972f4a5d9b924258271ee3c6.dblock.zip.aes (81.54 KB),
2019-05-09 01:00:04 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b2164a710972f4a5d9b924258271ee3c6.dblock.zip.aes (81.54 KB),

]
Warnings:
Errors: