I went to restore from a backup over the weekend and found that even though there was a successful backup on the 28th of June, it is not showing up in the restore list. Log shows files and everything.
Am I missing something?
Hello and welcome to the forum!
Is it possible no files were changed between the backup on the 27th and the one on the 28th? If no data is changed, Duplicati will not retain that backup (as there is no point).
If that isn’t likely, then let us know what your retention policy is set to.
Log shows 77 MBs uploaded and a list of several blocks “put”.
Using the “Smart backup retention” option.
Smart backup retention is the same as custom set to 7D:1D,4W:1W,12M:1M
If you are running only one backup per day, it is possible that they get pruned a bit early if slightly less than one day elapsed since the previous backup.
I suggest you use custom retention and set it to 7D:U,4W:1W,12M:1M - this will retain ALL (U=unlimited) backups performed in the past 7 days. Also if you want to keep backups longer than 12 months, it should be adjusted further.
Personally I use “custom” set to 7D:U,3M:1D,2Y:1W,99Y:1M . That basically means keep every backup for the most recent 7 days, keep daily backups for 3 months, keep weekly backups for 2 years, and keep monthly backups for 99 years. (I don’t want it to delete old backups…)
I changed the retention to what you suggested (unlimited dailies) but I do not understand why that matters?
There was only one backup a day.
The key idea is you probably referred to calendar day. Duplicati sees a day as a precise 24 hour interval, computed to the second. The minimum interval is relative to prior backup, not to any “calendar” concept.
“Smart backup retention” thins backups out to one per day (with exemption for backup that just finished).
Perhaps if you open the backup from the 29th you will see it deleted the 28th. Here’s an example display:
1 day is defined as 24 hours, so backups run every 24 hours may time-jitter back and forth on deletions.
You might prefer to set a custom retention instead of smart retention (which is 1W:1D,4W:1W,12M:1M).
Changing the 1D to 23h (23 hours) would allow for some backup time variation before two are too close. Proposed solution of specifying U for unlimited will be equivalent unless you tend to do multiple/24 hours.
Before the backup schedule was regularized (I assume) at 2:00 PM, you can also see how the June 24 afternoon backup followed the June 23 late night backup by less than 24 hours. That one’s also missing, presumably deleted after the June 25 backup ran and noticed that June 24 was too close to the June 23.
Yep, just checked and it did delete the backup.
That seriously needs to be fixed. There are probably tons of people using that standard option and I highly doubt they want their dailies being deleted like this.
Backup unexpectedly deleted with custom retention plan and scheduled backup #3987
looks like relevant open issue. Fixes come from issues (limited by volunteers). Forum can’t track.
By linking to the issue, it updates, so may raise visibility. Feel free to read through and add a note.
Switching to calendar base is messy because interval can be non-calendar things, e.g. seconds.
There are also culture-varying definitions of even what it means to be per week. Time is just time.
This is likely a common and visible case, based on reports. There are few on weeks and months.
The fix may be a combination of documentation (manual, code) and somehow allowing tolerance.
If you look at the issue, you can see precedent for allowing tolerance, but it’s not done for this use.
Simplest “fix” for this particular case might be either a fixed allowance, or tweak “smart retention”.
Could you look at your June 27 backup log to see what second it started at? They’re usually close.
That’s not always good enough when the backup schedule and the interval are precisely identical.
is “Smart backup retention”. It could possibly be altered a bit, as was suggested for custom values.
Maybe take a few seconds off (how far off was yours?). Or just give a fixed tolerance for them all…
That will solve some problems of colliding values for typical use. Documentation can cover experts.
The logs and the web results all show the same exact second.
I read through the bug comments to a point but the issue seems convoluted and I do not understand why it is such an issue even after reading through that bug thread.
If you are creating backups daily, weekly, or monthly, than the retention policy should be comparing the day, week or month, not the precise time.
Precise time should only be compared if more than one backup is found to share the same day, week or month, at which point newest would win.
One might not be, unless you’re asking for variable behavior depending on which option one selects.
The custom setting will let user set any arbitrary interval that they prefer, whether ot not you prefer to.
Maybe that’s too flexible, but it was there before the thinning-with-age retention feature was added in.
As mentioned (and linked), weeks start on various days, which might be Saturday, Sunday, or Monday.
It gets much worse the further one gets into calendar. If you would like see “convoluted”, look this over:
and after that, look left at the other topics on the bar. Looks hugely more complex than using intervals.
Maybe it simplifies to something easier than I think, but at least the feature developer didn’t go there…
Some of the original discussions are below. Things such as “week” variations came up several times.
Staggered versioning / thinning out versions
Retention Policy like Time Machine [$50] #2084
New retention policy deletes old backups in a smart way
While I agree something “should” be done to reduce accidents, I’m not sure that “calendar” is the way.
With a very large backlog of issues and requests, development must keep development effort in mind.