Daylight saving shifts job starting times?

Support
1

I just was out of town for three weeks and my backups ran fine and without errors. Yay! :slight_smile:

Today i noticed an odd thing: my timeplan shifted an hour. I just can assume this happend due daylight saving changes here in europe. Could this be? A Bug perhaps?

What happened on my wndows machine is: i do have 10 backup jobs starting daily at 08:xx and being repeated every 8 hrs. Now they 're are all set to 09:xx.
The DST changed on my machine correctly.
I did now manually set the starting times for the jobs back to 08:xx. But this should not happen, should it?

1 Like
2

Yep, it’s a bug :frowning:

DST changes scheduled backup time · Issue #1364 · duplicati/duplicati · GitHub

3

Ah, ok. Seem’s to be around for a while :wink:
It’s no show stopper in my case, but good to know that it is not supposed to be like that.

Thanks :slight_smile:

4

Not gonna lie: was disappointed this bug did not get fixed in the new beta

1 Like
5

As I say many times a day here (or at least so it seems), volunteers are extremely scarce. Any takers?
Rewriting a scheduler as proposed by original author here sounds daunting for someone not so expert.

There are also questions (in my view) of how to handle times that don’t happen, or that happen twice…

You can search for “List of backup timestamps contains duplicates” to see the mess that was causing.

was the fix. I later noticed another old worry of mine here, concerning how to handle ambiguous times:
List of backup timestamps contains duplicates #2857 shows worries about code Duplicati runs under.

Time Zones Processing is WRONG is one (yours) where day-of-week was being computed with UTC.
Use local time when checking if day is allowed #3528 solved that, and you can again find a suggestion
that the scheduled times be stored in local time, which seems to invite other trouble. A specific design:

which in itself didn’t seem a great use of scarce resources, did at least get me to see how cron solves:

Daylight Saving Time and other time changes

Local time changes of less than three hours, such as those caused by the start or end of Daylight Saving Time, are handled specially. This only applies to jobs that run at a specific time and jobs that are run with a granularity greater than one hour. Jobs that run more frequently are scheduled normally.

If time has moved forward, those jobs that would have run in the interval that has been skipped will be run immediately. Conversely, if time has moved backward, care is taken to avoid running jobs twice.

Time changes of more than 3 hours are considered to be corrections to the clock or timezone, and the new time is used immediately.

It's possible to use different time zones for cron tables. More could be found in crontab(5).

Because there have been at least two fixes that didn’t require a major redesign, I hope there’s an equally simple way out of this annoying time shift. Someone would have to look, and there are VERY few people.
Having seen the complex cron scheme, do you have a proposal for how a minimal solution should work?

1 Like
6

Didn’t look at this but there has to be a code differential like looking to see if daylight has occurred. There should be some way of getting that true or false value like in .net TimeZoneInfo.IsDaylightSavingTime Method (System) | Microsoft Docs

Once one has that and the old value to compare, one just has to decide what to do that knowledge. There’s probably multiple ways it could be incorporated.

But, if timestamps have major problems then what about someone traveling and the backups run but the time has 12 hours of repeat. It has to be able to deal with that if time is used in such a way or time should be removed from the equation.

That would be a major issue really if everyone’s backups broke because the CMOS battery died lol.

But if the backups continue to run and the time is off, at least it didn’t have a meltdown :slight_smile:

7

Castigating people for not volunteering skills they may or may not possess is… one way to get more volunteers. Probably not one of the more effective ones, however.

The cron strategy for DST change handling is brilliant enough to seem obvious. That’s how things like that work: hard to come up with but obvs once we see them. I think it’s the way to go. If I am soon blessed with the free time needed I will indeed pick it up and do it.

One thing’s for sure a self-inflicted wound: the system works internally in UTC but displays everything in the local time zone choice. This does not become obvious until you’ve scheduled something that suddenly jumps to a different day. Having things in your own local framework jump around by an hour in either direction seems weird to me. Even if they don’t cross a day boundary – that just increases the havoc.

If I schedule it in a tz it should stay put wrt that tz. If I’m really scheduling it in UTC it should only ever display in UTC, I guess. If you tell me that it’s weird to show people their schedule in a different tz than they work in, I ask you, why are you internally scheduling things in a different tz than I work in? Why are you scheduling things to ignore DST when I do not?

It’s especially troublesome when I have things like replication jobs scheduled via cron, which does not misbehave about time zones. I have to put extra hours of buffer time in case I forget, Oh, DST tonight; duplicati will have its semi-annual seizure.

8

What would be effective? If there is none, then project proceeds at the pace that volunteers can move it.
If volunteers get too low, project stagnates even more than it’s currently stagnating (relative to requests).

Thanks to @Xavron for starting to assist. Presumably it followed reading forum. I read for quite awhile. Everybody starts somewhere, and some ramps are harder. Even current developers are still learning…

I haven’t done a recent study, but I’m pretty sure I’ve seen advice saying that this is the best way to do it, from a programming point of view. From a user point of view, time jumps at DST regardless of the code.

I did a quick Google search for “best practice” “time zone” “UTC” “local” “user interface” to get thoughts.

Not answering the corner case question. If, say, DST happens at 2:00 A.M., and you schedule for 2:30 in Spring (forward), is it OK to skip when 2:30 doesn’t happen (because local times jumps to 3)? In Autumn, schedule for 1:30 potentially happens twice. For a first cut, would such oddities be acceptable? This may possibly not require rewriting the whole scheduler. Big job, and I’m not sure Canary test can cover it well, because there aren’t that many users, DST change is only twice a year, and users might expect oddities.

If it has to get fancier like cron does it, then it gets harder but might still be simpler than scheduler rewrite.

If you look at the GUI on your backup, I believe you see the next run is scheduled when current run begins. This seems like a good time for code to set a proper next UTC after considering the plan for DST change. The rest of the code can maybe stay as-is, doing UTC-to-local conversion for “Next scheduled task” and kicking off jobs in UTC as currently done. I’m writing this from memory, but this might simplify the change.

What’s interesting is that the times in Duplicati-server.sqlite (which are UTC) get updated after the backup which is different than what the GUI job’s “Next scheduled run” does. I checked – it did update at run start.
I’m not sure what to make of this difference, but what I wonder about is – can smart next run time fix this?

is possibly in the same neighborhood except it sounds like a later check, and I’m talking about in advance.

There seems to be at least a bit of developer expertise at how the run times get set, because this is fixed:

Fix issue where multiple backups could occur after pause #4180 (which also leads to the scheduler code)

although it does mention LastRun time, and I’m probably talking Time at least in DB Schedule table name.

I think this would make the UX worse all the time by many hours instead of only weird from the DST shifts. Please have a closer look at the “best practice” articles I cited, and see what you think. I have some UNIX experience, and been very used to doing the internals in Epoch time, so my view might be colored by that.

That’d be great. Not pointing at you specifically about volunteers, but it really is a very serious need to even come close to meeting the expectations of the increasing number of users who are asking for things done.

1 Like
9

I think this would make the UX worse all the time by many hours instead of only weird from the DST shifts.

Yes it would make the UX worse to display UTC.

But you seem to be baffled by the fact that all around you, there are 1,000 different systems that schedule events to particular times of the day, and 999 of them do not shift the local time of scheduled events when DST begins or ends. 999 of 1,000 have penetrated the deep software engineering mystery of what to do when DST toggles. One and only one has not.

That’s right: duplicati is the only software I have ever encountered in my 42-year IT career where a 6PM backup becomes a 7PM backup on the second Sunday of March… all within the confines of one machine, in one time zone.

Please stop pretending to treat this as a bleeding edge mystery of software engineering. Admit that duplicati screwed the pooch on this one (small) issue. It should be prioritized to be fixed after the years and years it’s been going on. The resource problem is real, but the resources are not zero.

10

Guess you never used Symantec (Veritas) Backup Exec! It had this same annoying issue for a while. Other than that and Duplicati, I can’t think of any other examples. (I’ve been in IT for 30 years.) So yeah you make a great point.

I think everyone agrees this needs to be fixed. It just needs some attention from developers, and resources are kind of thin.

1 Like
11

Actually I have worked in shops that used Veritas, but not as a member of the BUR team. I guess they shielded the rest of us from it. Probably set reminders to fix it on two certain Sunday mornings of each year :wink: And of course, that bug is no longer in Veritas - hmmmmmm.

I also remember people occasionally modifying job schedules to move things out of the “witching hour” between 02:00-03:00 in the spring or 01:00-02:00 in the fall.

12

I did, in fact answer the “corner case question.” Do it the way cron does it, full stop. Run a job at 3AM on that day in the spring if it would have run normally at 2:30. Run a job that normally runs at 1:30 once only on that day in the fall, despite the fact that you get two 1:30s.

How about you answer the main - not corner case - question which is, why does DST toggle permanently move all job schedules by an hour (absent human intervention) for the 363 days when DST is NOT changing?

13

DST changes scheduled backup time · Issue #1364 · duplicati/duplicati · GitHub dates back to April 2015.

It’s time.

1 Like
16

Yes you did. Apologies.

If you pick it up, you can consider (probably after code study) whether or not going “full cron” is best, e.g.

If time has moved forward, those jobs that would have run in the interval that has been skipped will be run immediately.

sounds to me (and you can confirm – you have more cron skill) that you get a burst of jobs in the Spring.

1:45 job every 30 minutes
1:45 AM
2:15 AM doesn't happen
3:00 AM roughly 15 minutes from previous
3:15 AM roughly 15 minutes from previous
3:45 AM

I’m proposing that the chosen interval be maintained, basically just pick the right UTC for the next time.

1:30 job every 30 minutes
1:30 AM
3:00 AM
3:30 AM

Duplicati does have a missed-job practice of ASAP run when Duplicati starts. Maybe it could be used,
but I kind of think staying on requested interval is better. I don’t administer systems. Feedback sought.

Of course almost anything other than the current plan is an improvement, so where it lands may vary.

Scheduling strategy for backups has design information from the original author to credit for Duplicati.
Everybody wishes it were better than it is (ever getting out of Beta?), but the resources are super thin.

I didn’t start the project, I don’t run it (but I do have a little bit of say), and I wouldn’t be writing the code.

17

Spring is easier. 1:30, 3:00 is exactly the sequence I’d expect when the hour from 2:00 to 3:00 (well, 2:59:59) is removed.

In the fall, it may require asking the user what they want. I think it’s only germane if the user has jobs scheduled at tighter intervals than one hour. At wider intervals, there might be one job that would be repeated. The task is simply to suppress that repeat and then move on with life.

If there are jobs at tighter intervals: beginning a week before, on the schedule screen, I would offer the user a projection of all the jobs that might run around that time, and allow them to check/uncheck the ones they want to run. If the user never visits the schedule screen, I would fall back to a default behavior.

The default behavior would be: take the normal schedule and uncheck any that would start in the second instance of 1:00-1:59:59.

18

When the time skip is caused by DST change and not a duplicati outage, I think it’s also a case of, what are the intervals between jobs. In this case, I think if the interval is MORE than one hour, the default behavior is correct. The canonical example is a backup scheduled every day at 2:30. On DST spring Sunday that should run at 3:00 as a one-off. Its scheduling will remain, of course, 2:30.

If jobs are scheduled every 15 minutes at :05, :20, :35, and :50, then I would not run at 3:00, but wait for the 3:05.

19

I prefer keeping it simple, because making it ambitious decreases the chances it ever gets done.
My hope is this can be fixed by working out where next job should go, then find code and a coder.
I’m sort of counting on adequate library help, e.g. so we can know when next DST change will be.
I’m pretty sure the info exists, but I’m less sure about how easily the libraries will allow querying…

Crontab basic syntax (e.g. */2) seems to allow every-other-hour jobs, but they run at even hours:

12 AM
2 AM doesn’t happen, because it goes to 3
3 AM “run immediately” rule applies, although actually it’s still only 2 hours from the previous run.
4 AM run because we’re back on even hours.

I think cron allows minute schedule offsets. On Duplicati, base can be anything, then set interval.
I’m not sure how much the philosophy differences will interfere, but I guess we’ll see as we go…

Duplicati would have to note that in a DST time change situation, the specified backup gap varies.
Keeping it the same might please some people, but a steady 24 hour gap means clock-time shift.

Although I’m sure cron has had global exposure, one thing to watch out for is that the DST shift is
not always one hour, however https://www.timeanddate.com/time/dst/ says it’s better than in past.

Regardless, the time functions that get used have to worry about country specific policies anyway.
Time Zone Database has the raw material that the library algorithms would use for Duplicati’s use.

I’m hoping that leap seconds don’t get into this, but we’ll see.

The cron text seems to say (again, I’m reading not testing) says for special rules, “This only applies to jobs that run at a specific time and jobs that are run with a granularity greater than one hour.”, so jobs running at exactly one hour interval get usual handling in cron. While it should be tested, I’d guess Duplicati would run

1 AM with next backup 60 minutes later hits
2 AM on time before Spring jump but is now
3 AM so again that gap looks like 2 hours but is actually 1 hour, and this is probably as good as it can get…
4 AM

Reading the cron text again, they seem to have special handling for “run at a specific time” and also for “run with a granularity than one hour”. Duplicati doesn’t exactly have specific times, other than intervals.

Would your example be “specific time”? I’m not even sure this concept fits Duplicati, but intervals do…

That’s like what I was aiming for with the 1:45 example I began, but inadvertently drifted. I meant to show:

1:45 job every 30 minutes
1:45 AM run
2:15 AM doesn't happen
2:45 AM doesn't happen
3:00 AM happens but we don't do catch-up because interval is not greater than 1 hour
3:15 AM run
20

There’s a DST change again, and all jobs shifted …

Can Duplicati get a general option that chooses between (all times) use of UTC or Local Time? I know nothing about the internals but I think this will suit all uses.

Hoping it’s present in Spring 2022 … for such an advanced tool like Duplicati this regular manual change of jobtimes is just silly.

21

Export all your jobs as command-line and put them in cron (or AT if you’re on Windows)

Waiting for this to get fixed in the software is the very definition of insanity: doing the same thing over & over, and expecting a different result.

22

If you do it that way, note that it operates independently from the Web UI. You will not see job progress or status there. If that’s important to you, I recommend using the excellent duplicati-client command line utility. With it you can trigger Web UI jobs to run.

1 Like