Retention Versions Missing

Support
1

Can’t decide if this is a bug or a just a mis-understanding.
I’m trying to backup my system using smart versioning. The scheme that I’m looking at is a version each day for the last 7 days, then 1 for each week for the previous 4 weeks(month), and then one a month for the last year.
In my backup retention, I have the following custom retention set: “1W:1D,1M:1W,1Y:1M”

I’ve been backing up my system for a couple weeks now, and when I look at my available versions, I seem to be missing some from the current week:
image

Looking at the messages for the backups from those missing days shows the backup running, and I think it was successful.

Thoughts?

2

Your retention rule looks correct to me for the stated goal.

So in your specific case you think you should have restorable versions for Apr 30, Apr 31, and May 2, right?

Can you either post the “Result” messages for those days or check in them for ParsedResult lines? If you’re using a more recent version of Duplicati you should find multiple lines in sections such as:

  • DeletedResults - Was deletion of destination files successful or not?
  • BackendStatistics - Was fetching destination file info successful or not?
  • TestResults - Was testing of destination files successful or not?
  • “Main” section (no indentation) - was the overall job successful or not?

Oh, and pay attention to the date of the log entry (or EndTime in the log) - it’s possible your backups either ran so long they ended in another day or the machine was asleep at 1:00 AM and the job didn’t run until the machine came back online which could have been the following day (thus causing two backups on one day to would be cleaned up by the 1W:1D rule).

3

Attached the results logs for Apr 30 - May 3. Nothing I could really see to show why the versions for Apr 30 and May 2 are missing.

Also as an aside, I’m not sure how it figured out which revision to keep for the 1M and 1Y setup. Ideally, what I’d really want:

  • A rolling 7 days
  • The first backup taken after Sunday at midnight each week for 4 weeks.
  • The first backup taken each month for 12 months.
    I haven’t had it running long enough to test the monthly retention in particular, but given that I only have versions from the 21 (my first run), and the 27th (just checked now, and it’s removed the 27th and kept the 28th), I’m not sure how it determined those ones to keep.
    DuplicatiLogs.zip (2.0 KB)
4

Could it be (by any chance ;-)), that on the missing days absolutely nothing changed on both of your sources?
It happens sometimes on my sources too and the backups that ran, but found nothing changed, don’t show up under restore…

5

Is it possible there were no files changed on the “missing” days? If so but you want a backup run to show in the restore list anyway, try adding --upload-unchanged-backups=true to your job:

--upload-unchanged-backups
If no files have changed, Duplicati will not upload a backup set. If the backup data is used to verify that a backup was executed, this option will make Duplicati upload a backupset even if it is empty
Default value: “false”

You can double check that by looking at the Result log for the “missing” days. In my logs below you’ll see that May 3 is “Missing” but if I check the log I the following which indicates that there were just no changes to be backed up.

DeletedFiles: 0
DeletedFolders: 0
ModifiedFiles: 0
...
AddedFiles: 0
...
AddedFolders: 0
...
ParsedResult: Success
...

I don’t recall exactly but I think it keeps the MOST RECENT version in any particular time frame.

I’m not sure what 1W:1D would do with “TODAY’S” backups if doing them hourly (would it keep them all until tomorrow or only keep this hour’s backup and delete the earlier ones) so I tend include 1D:U at the beginning of my rules. This also helps cover multiple manual runs I might be doing when testing / changing jobs.

I’m also unclear as to whether it treats W as a rolling 7 days, Mon-Sun, Sun-Sat, etc. (different countries define “week” differently) so I tend do use 7D instead of 1W.

Note that there are other edge cases to consider such as week / month / year overlaps in the sense that Year might end in the middle of a week and week might end in the middle of a month.

If you want some historical examples to review, here is my restores history with 1D:U,1W:U,1M:1D,13M:1W,U:1M job on an always-on server (so not sleep issues). (Note that some were manual runs and there were some tweaks of run times.)

As you can see I’m using W and M periods against what I suggested above (I guess it’s been a while since I’ve updated this job) and the actual days of the week that things run are a bit dynamic.

Latest:

  • May 4, 2018 3:22 AM (Fri)
    This week:
  • May 2, 2018 3:22 AM (Wed)
  • May 1, 2018 3:22 AM (Tue)
    This month
  • Apr 26, 2018 3:22 AM (Thu)
  • Apr 24, 2018 3:22 AM (Tue)
  • Apr 23, 2018 3:22 AM (Mon)
  • Apr 22, 2018 3:22 AM (Sun)
  • Apr 21, 2018 3:22 AM (Sat)
  • Apr 20, 2018 3:22 AM (Fri)
  • Apr 19, 2018 3:22 AM (Thu)
  • Apr 15, 2018 3:22 AM (Sun)
  • Apr 14, 2018 3:22 AM (Sat)
  • Apr 12, 2018 3:22 AM (Thu)
  • Apr 11, 2018 3:22 AM (Wed)
  • Apr 10, 2018 3:22 AM (Tue)
  • Apr 9, 2018 3:22 AM (Mon)
  • Apr 8, 2018 3:22 AM (Sun)
  • Apr 7, 2018 3:22 AM (Sat)
  • Apr 6, 2018 3:22 AM (Fri)
  • Apr 5, 2018 3:22 AM (Thu)
    Last month:
  • Apr 3, 2018 3:22 AM (Tue)
  • Mar 27, 2018 3:22 AM (Tue)
  • Mar 19, 2018 3:22 AM (Mon)
  • Mar 11, 2018 3:22 AM (Sun)
  • Mar 4, 2018 2:22 AM (Sun)
    2018
  • Feb 24, 2018 2:22 AM (Sat)
  • Feb 17, 2018 2:22 AM (Sat)
  • Feb 10, 2018 2:22 AM (Sat)
  • Feb 2, 2018 2:22 AM (Fri)
  • Jan 26, 2018 2:22 AM (Fri)
  • Jan 18, 2018 2:22 AM (Thu)
  • Jan 11, 2018 2:22 AM (Thu)
  • Jan 4, 2018 2:22 AM (Thu)
    2017
  • Dec 28, 2017 2:22 AM (Thu)
  • Dec 21, 2017 2:22 AM (Thu)
  • Dec 14, 2017 2:22 AM (Thu)
  • Dec 7, 2017 2:22 AM (Thu)
  • Nov 29, 2017 12:00 PM (Wed)
  • Nov 22, 2017 12:00 PM (Wed)
  • Nov 15, 2017 12:00 PM (Wed)
  • Nov 7, 2017 12:00 PM (Tue)
  • Oct 30, 2017 1:00 PM (Mon)
  • Oct 23, 2017 1:00 PM (Mon)
  • Oct 16, 2017 1:00 PM (Mon)
  • Oct 9, 2017 11:31 AM (Mon)
  • Sep 21, 2017 1:00 PM (Thu)
  • Sep 13, 2017 2:06 PM (Wed)
6

Thanks guys. Definitely possible that nothing changed for those missing days. The log for May 2 does show that:

May 2, 2018 1:01 AM: Result
DeletedFiles: 0
DeletedFolders: 0
ModifiedFiles: 0
ExaminedFiles: 95574
OpenedFiles: 0
AddedFiles: 0
SizeOfModifiedFiles: 0
SizeOfAddedFiles: 0
SizeOfExaminedFiles: 156458902841
SizeOfOpenedFiles: 0
NotProcessedFiles: 0
AddedFolders: 0
TooLargeFiles: 0
FilesWithError: 0
ModifiedFolders: 0
ModifiedSymlinks: 0
AddedSymlinks: 0
DeletedSymlinks: 0

However, the log for Apr 30 does show some changed data, so not sure why that’s missing.

Apr 30, 2018 1:01 AM: Result
DeletedFiles: 1
DeletedFolders: 0
ModifiedFiles: 42
ExaminedFiles: 95563
OpenedFiles: 56
AddedFiles: 14

I’ll try adding the --upload-unchanged-backups=true argument to my job.
As for the retention policy itself, your logs do show some interesting results, but it still seems to make it hard to predict exactly which ones will be kept. Also, I’m going to assume the U qualifier either means Unlimited or Unscheduled depending on whether it’s in the first or last position (ie. 1D:U vs U:1M)

7

The Rentention Policy algorithm is fairly simple:

  • Always keeps the most current backup
  • Sort each backup into the corresponding timeframe
  • For each timeframe do
    • Always keep the oldest backup in the time frame
    • For each backup in the timeframe do
      • If the time difference to the last kept backup in the timeframe is smaller than the specified intervall, delete it.

So if you have a rule 1W:1D and run backups hourly, it will keep the backup that was just created and then delete the following (second) backup from the hour before, if it is less then 86400 seconds (=1 day) away from the third backup.

Yes, every timeframe is rolling, for the reason you mentioned (different countries define “week” differently) and since the user can theoretically specify unusual timeframes like 4D.

As to why Caldorians backups were delete:

May 2nd: According to the logs, there were no changed or deleted files as JonMikeIV suspected, so there simply was no backup created

Apr 30th: This one is a bit mysterious tbh … If it got deleted by the retention policy, then it must have been less then 86400 seconds (1D) away from the backup on the Apr 29th. @Caldorian Do you happen to have the logs for the Apr 29th backup as well?

1 Like
8

Here’s the log from the 29th:

DeletedFiles: 0
DeletedFolders: 0
ModifiedFiles: 1
ExaminedFiles: 95550
OpenedFiles: 2
AddedFiles: 1
SizeOfModifiedFiles: 26615
SizeOfAddedFiles: 308098
SizeOfExaminedFiles: 156444819442
SizeOfOpenedFiles: 334713
NotProcessedFiles: 0
AddedFolders: 0
TooLargeFiles: 0
FilesWithError: 0
ModifiedFolders: 0
ModifiedSymlinks: 0
AddedSymlinks: 0
DeletedSymlinks: 0
PartialBackup: False
Dryrun: False
MainOperation: Backup
CompactResults:
DeletedFileCount: 0
DownloadedFileCount: 0
UploadedFileCount: 0
DeletedFileSize: 0
DownloadedFileSize: 0
UploadedFileSize: 0
Dryrun: False
MainOperation: Compact
ParsedResult: Success
EndTime: 2018-04-29 1:01:09 AM (1524978069)
BeginTime: 2018-04-29 1:01:05 AM (1524978065)
Duration: 00:00:04.5638810
BackendStatistics:
RemoteCalls: 8
BytesUploaded: 9745096
BytesDownloaded: 61813503
FilesUploaded: 3
FilesDownloaded: 3
FilesDeleted: 0
FoldersCreated: 0
RetryAttempts: 0
UnknownFileSize: 0
UnknownFileCount: 0
KnownFileCount: 4488
KnownFileSize: 117192686549
LastBackupDate: 2018-04-29 1:00:01 AM (1524978001)
BackupListCount: 6
TotalQuotaSpace: 10638670745600
FreeQuotaSpace: 3849542443008
AssignedQuotaSpace: -1
ReportedQuotaError: False
ReportedQuotaWarning: False
ParsedResult: Success
DeleteResults:
DeletedSets:
Dryrun: False
MainOperation: Delete
ParsedResult: Success
EndTime: 2018-04-29 1:01:09 AM (1524978069)
BeginTime: 2018-04-29 1:00:58 AM (1524978058)
Duration: 00:00:10.9410038
RepairResults: null
TestResults:
MainOperation: Test
Verifications: [
Key: duplicati-20180429T050001Z.dlist.zip
Value: ,
Key: duplicati-i5baa363f6ac54fbebbcdb098696a9b69.dindex.zip
Value: ,
Key: duplicati-bc18aceb8653c43a7af47846f4246d55e.dblock.zip
Value:
]
ParsedResult: Success
EndTime: 2018-04-29 1:01:22 AM (1524978082)
BeginTime: 2018-04-29 1:01:21 AM (1524978081)
Duration: 00:00:00.9313946
ParsedResult: Success
EndTime: 2018-04-29 1:01:22 AM (1524978082)
BeginTime: 2018-04-29 1:00:00 AM (1524978000)
Duration: 00:01:22.0622757
Messages: [
No remote filesets were deleted,
Compacting not required
]
Warnings:
Errors:

That said, given the context of how you said the policy is calculated, it seems like there’s a fatal flaw in it. I would assume that if I set a retention of something like 30D:1D, with backups scheduled for the same time every day, then I’d have a rolling set of 30 daily backups. But your indication that it’s looking for minimal time between doesn’t allow for any type of drift in processing times, and eventually, you’ll end up with backups missing because there’s going to be a small overlap.

9

Yep exactly, and that’s what happened in your case. According to your logs, the backup from the 29th was created at 1:00:01 AM, so the backup from the 30th at 1:00:00 AM was deleted, because the time difference was a second too short to reach a whole day (86399 instead of the needed 86400)

I agree that there should be a tiny amount of wiggle room to account for such cases, though the question is, how much? If it’s too much (e. g. several minutes), then it’s more and more likely that it could potentially lead to cases where you suddenly end up with two backups for a day (e.g. 00:00:05 and 23:59:43). That might be confusing as well, cause the expectation might be, that 1D will keep at max one backup per day.

So my completely arbitrary suggestion would be 10 seconds for this wiggle room …

Or do maybe you have any other suggestions or even better ideas how to deal with this?

10

Think it could easily be more. If my backups take anywhere from 3 to 15 minutes to complete before the 2nd job can run then 10 seconds won’t matter.

11

Don’t know if the retention options support it, but I guess you’d do something like 7D:23h instead (assuming you’re only backing up daily). Use a lowercase m if you want minutes.

However, I prefer something like what Avamar does. It’ll tag a backup when it’s the first one of the day/week/month/year and keep it around for the specified retention. Day starts at midnight of the time zone of the system, and I think week is configurable to pick your start point. Only scheduled backups are tagged by default, though you can specify a manual run to get tagged if you wish.

12

I don’t suppose 7D:86410s would work to give that 10 second wiggle room? :slight_smile:

13

The other way arround. You’d decrease the minimum interval. So 7D:86390s for 10 seconds or 7:23h if you want an hour of wiggle room.

I’m aware though, that this is a bit annoying and Pectojin is right with pointing out, that other backups that are still running might delay it way more than 10 second. Sadly, I don’t have a solution for it at the moment. I’ve spend the last two evening trying to come up with another approach, but … that one has some other drawbacks, which I’m also not too happy about.

14

I suppose some people would want a hard rule, even if it meant losing a few days due to delayed start times, while others would want a soft (or at least fuzzy) one, even if it meant having multiple backups appear in a single day DESPITE the rule in question.

Would it be enough to let the user decide if a rule should be hard (based on run time) or soft (based on SCHEDULED time) with appropriate commenting added to description / docs?

Speaking of docs, at this point it might make sense to cut down a little on the in-GUI description and just include a link to the online docs where we could provide better examples… though I suppose that would be in all English… (Say “hello” to Google Translate?)

Alternatively, I wonder if it would be easier for people to understand if the UX produced a sentence summary of the rules similar to what is in the [Retention Policy] logs:

[Retention Policy]: Time frames and intervals pairs: 1.00:00:00 / Keep all, 7.00:00:00 / 1.00:00:00, 28.00:00:00 / 7.00:00:00, 365.00:00:00 / 31.00:00:100:

But more like:

For 1 day, keep all. Then for 7 days, keep 1 per day. Then for 28 days, keep 1 per week. Finally for 365 days, 1 per month.

Just don’t ask me to have to code it. :blush:

15

But I think that’s the issue: what does keep 1 per day/week/month mean? When you say 1 per day, I don’t think a minimum of 24 hours between any 2 backups. I think one for each date. Same with 1 per month. Week can be more ambiguous, but I’d generally think it would still be a defined 7 day period, with a fixed day of the week marking the start.

16

Very good point and a perfect example of how easy it is to go wrong with this topic. :slight_smile:

So to better match actual functionality it might make more sense to “sentencize” it like:

For 1 day, keep all. Then for 7 days, keep no more than per 24 hour period. Then for 28 days, keep no more than 1 per 7 day period. Finally for 365 days, keep no more than 1 per 30 day period.