I am sure this may have already been discussed here but I cannot find anywhere any guidance on how to setup backup hour intervals for a particular backup task.
For example I want a backup task with the following configuration:
Start everyday at 11pm and continue for 8 hours maximum (or until 7am next morning)
Take backups to S3 compliant off-site storage
If the backup task is completed within the time interval, then it should start a new backup
If the backup task is not completed within the time interval, then it should skip the new backup and resume the previous backup next morning
I have slow upload speeds and lots of files, so it will probably take weeks (if not months) to do the initial off-site backup. Thus, I need a solution to do backups at night and resume the next day.
Hi Beko, there is no mechanism to stop a running backup that takes more than X amount of time. It is true that initial backups can take a while. If your goal is to minimize how it affects your bandwidth during the day, then you can experiment with throttling options. You can’t have different throttling options at night vs at day though.
A more robust way to handle this is at your router by using QoS. That’s personally what I do - deprioritize backup traffic so that it consumes as much as it needs but will be scaled back automatically by other, non-backup traffic.
Unless your download runs faster, is weeks (if not months) acceptable for restore in case of disaster?
Also on the speed topic, I think throttling is confused, so if upload throttle doesn’t throttle the uploads, setting the same value for download throttle might solve it. Direction confusion may vary with version.
@drwtsn32 Thank you for your responses. We have a firewall and I assume we can set some QoS rules there, that is probably what I will try… It would be a great feature if Duplicati offered hour intervals in future updates though.
@ts678 Our dowload is 50x times faster than our upload, so restoring would take days instead of months. Also the point is not to use our limited upload during the day so we can continue working normally. If I cannot do this with firewall QoS, I will look into the bandwith limiter tools that you mentioned. Thanks!
Current state of scheduling is that Duplicati can’t even schedule its backups right per the local time zone. Instead there’s a shift. At least some more critical scheduling bugs have been fixed, but this one has not. Scheduling is hard, especially around time oddities, e.g. when certain local times repeat or don’t happen.
I just want to jump onto this too, I would also like to see the feature/option.
In my case, even during the incremental updates, sometimes many files (example exif of pictures updated) were changed that the new incremental backup takes quite some time.
I want to run backups over night, even though QoS is in place at source and backup destination, which lets people working from home on both location without much of a headache
I know, I could extend the ISP bandwidth for this, but this is not how I like to solve it.
About restore, most of the time I do only restore single files, which is fast enough with the 10 times faster download I have. If I need a full backup, I just grab the disk from my friends house where the backup location is.
My main issue with a backup taking longer is that the other backup sets are “stalled” until I stop the long-run-backup manually or the backup is finished (after days).
So defining a maximum runtime for a backup would allow the other sets to run and finish instead of being blocked by the one which is taking much longer then normal (for above mentioned reasons).
I use Duplicati in most cases on Synology. For such cases (hugh backups which can take the whole night), I created multiple docker containers, I just change the port for the containers.
As Duplicati never reaches the maximum transfer speed of the internet connection (even with compression level set to 1 and and so on, even on Synology machines with Xeon D-1527 CPUs, transfer speeds only reach 3-4MB/s at a maximum), there is no problem running 2 or 3 three Duplicati containers on the same Synology.
So everything fits in the nightly backup time frame…
where the backup location is.