Run backup when PC is idle

Crashplan has functionality whereby you can set more frequent backup intervals, but where a pending backup job will wait until the system has been idle for X number of minutes - implying that the user is AFK and therefore a running backup wouldn’t be disruptive.

This way I could set a backup to run every couple of hours on my work PC and be reassured that it’ll back up at least a few times a day, instead of the current functionality where I need to attempt to time it to run during my lunchbreak (even though i’m never sure when it’ll really be), etc.

1 Like

Thanks for the consideration, but it’s better to keep different features in separate topics so I moved your post to a new topic (on this forum threads are called topics because, as you probably noticed, discussions are not strictly threaded). Feel free to modify the title if you like.

Thanks for the clarification - I’m good with your title, though I edited my text (as you probably noticed) to account for no longer borrowing someone else’s thread :wink:

1 Like

Not sure if this thread is a better place to ask vs my original thread, but do you expect any DB issues with Duplicati having to handle roughly 24 times the versions of a typical daily backup config?

So this ended up being a bit longer than I expected… Long story short: If you want to have more robust scheduling options, you could utilize the task scheduler included with your OS. If you have not used task scheduler before it may seem a little daunting; However, if you are using Duplicati, I am confident that you can work out the task scheduler as well! For a good starting point, I have written up a step-by-step guide to get a task set up to accomplish the requested goal of this topic. Expand the guide below to check it out!

Scheduled Task Guide - Run when idle

First, keep in mind the GUI is really just a wrapper. Backup commands work just as well when called directly from the command line and the GUI is capable of exporting the exact command you need to run. While you wait for this feature request for the GUI scheduler to go through, you can set up a Windows Scheduled Task to do exactly what you want to do! I created such a task to test this out and it seems to work for me. Here are the steps to set it up, I am assuming you are using Windows, but I’m sure you can adapt these methods to use any other OS’s scheduling functions.


Once you have your backup job tweaked and perfected in the GUI, select “Export …” from the Configuration group:

Choose “As Command-line” and push Export

This will generate the command line that the GUI calls when it runs a backup. In my case, I am using “Alternative FTP” to connect to my server using FTPS (this is my internel test ftp so my credentials are weak! You probably want stronger security if you are exposing this to the internet!), with email notifications, using 7-zip, all kinds of fancy stuff:

PLEASE NOTE! EVERYTHING WILL BE IN PLAINTEXT!!! This includes email server passwords and your backup’s encryption password, if you use backup encryption. There are methods you can use to obfuscate this information using Powershell or other scripting languages to wrap the command into a script or exe file, but that is beyond the scope of this post. For now, just make sure to lock your workstation when you walk away and utilize other methods to help secure access to the machine in the first place.

One important option that you should take note of is “dbpath”, which I have marked in the image above. I didn’t do anything special to make it show up so you should have it too. If you are using a service account to run Duplicati and you did not specify an alternate DB path in the Job’s settings (Under Advanced->Database…->Location), you will need to either set up the scheduled task to run under that service account’s context or grant yourself access to that location. In my example here, I use a service account to run Duplicati in the background so I just granted my myself (Username: Mono) access to my service account’s (Username: Duplicati) HOME folder so I can access the same database that the GUI uses.


Now you have everything you need for your scheduled task. Open Windows Task Scheduler and select “Create a New Task”, do not use “Create a Basic Task”. Give it the coolest name you can think of:

00

Under “Security Options” in the dialog above, make sure the user that is running the command has access to the dbpath you found earlier. I recommend enabling “Run with highest privileges” so the backup process is able to access all files, assuming you are an administrator. I also recommend leaving “Run only when user is logged on” selected initially because this will cause the command prompt window to show up while it is running. This will let you know the backup task is actually running and you won’t have to wonder if it is running when you experience a slowdown on your PC. If you set this username to a Service account, you will need to enable the other option and the command prompt will be hidden. This is fine once you tested it a few times under your own user context so you can make sure it is working as expected.


Next, go to the Triggers tab and push “New…” near the bottom. Matching your request, I created the task to start at 8 AM on every weekday, repeating every 2 hours until 12 hours later:

You probably want to disable all of the other options except “Enabled” until you get an idea of how the other options affect the task. For Duration, the drop down box presents only a few limited options. You may notice that this drop down box is also a text box! You can type whatever you want! just want it running during the workday? Type in “8 hours” or “480 minutes”, whatever you want really. The same goes for “Repeat task” if you want it to run every 1 hour or every 30 minutes. This is only when the task will start attempting to run; The “idle” checking will be done on the Conditions tab a couple screenshots down.


Next, click on the Actions tab and push “New…” near the bottom. This is where you will enter the command line you exported at the beginning of this adventure:

In the “Program/script” box you will copy and paste ONLY the EXE command which is conveniently in quotation marks already on your export (Don’t forget to include the quotation marks!):

27-2

In the “Add arguments (optional):” box you will copy and paste everything else following the exe command, starting with the word “backup” all the way to the end of the command. The text box is small but it will gladly accept the super long argument list. I set “Start in” to the Duplicati 2 folder containing the EXE file, copied from the command itself. I’m not sure this is required for Duplicati but I entered it to be on the safe side.


Next, go to the Conditions tab. This is where you let Windows know that you only want this running when the computer is idle. For this example, I set it to wait until the computer has been idle for 10 minutes:

30

I set “Wait for Idle” to 100 minutes. What this means is that Windows will wait up to 100 minutes for the computer to be idle. If the computer is not idle within the next 100 minutes, the current run will be cancelled. I set the time to be shortly before the repeat time on the first page (2 hours). Essentially, Windows will wait almost 2 hours for the idle window, after which it will give up. Since the task repeats every 2 hours, if you reach the 100 minute timeout period, it will try again 20 minutes later which would be 2 hours after the last time the task was triggered/attempted. I have not tested this through a whole day to make sure the timings are correct so you may need to tweak these settings to get it running correctly. I will update this post if I find that these settings are not correct. Make sure to disable all of the other options, especially “Stop if the computer ceases to be idle”, as that would cause your backup process to be immediately killed when you get back, requiring a repair of the database at the very least.

FINALLY, go to the Settings tab and make sure “Run task as soon as possible after a scheduled start is missed” is enabled. That way if you don’t turn your computer on or log in until after 8 AM, your backup cycle will still trigger and run through the day. I recommend disabling all remaining options:

29

Push OK and that’t it! Right click the task and select “Run” to test it out. Please note, selecting “Run” will NOT check for idle time and the task will run immediately.

There are a couple of caveats to keep in mind with this setup:

  • Since this scheduled task is using the exact same database as the GUI, try to avoid have the GUI schedule run at the same time that your windows task would run, or else you will get “Database is locked” errors. Personally, I would use the GUI schedule for a single nightly backup where you know you’ll be away, or just disable the GUI schedule completely and do you all of your scheduling using the Windows task scheduler.
  • If you change anything with the backup you will need to export to commandline again and manually update the Action under the “Actions” tab. Simply clear out the “Add Arguments” field and copy/paste the newly exported command arguments and that’s it. This includes seemingly minor changes to your Source Files and Exclusions listings or anything at all in the GUI.
  • Alternatively, you could place the whole exported command line into a BAT file and just run that BAT file as your “Action”. That way, you would only need to keep your batch file updated with the latest exported command line instead of needing to dig into your scheduled task any time you want to update your source files/retention time/volume size/whatever.
  • Backups performed outside of the GUI don’t update the “Overview” statistics for that job in the GUI, so it will not change the number of “versions available” as you make backups with task scheduler and maybe some of the other details. However, if you go to restore, you will see that all the backup versions made through the day are certainly available. I have found that running a backup with the GUI will update the overview to include everything done outside of the GUI, so maybe keeping a single nightly GUI scheduled run is not a bad idea to keep your GUI statistics somewhat accurate, if you’re into that sort of thing

This is a bit involved but totally worth it. Once you get the feel for task scheduler you can set up some complex backup schedules to do just about whatever you want. Good luck!

4 Likes

I’m not sure I understand the question, but it seems unrelated to both topics so the best way to get an answer is probably to create a new topic for it.

Generally, a good rule of thumb is that if you are not replying to any specific post in a topic, you probably want to create a new topic.

Hint: behind the :gear: icon in the editor, there is the “Hide details” option. Your post would be a good use-case for it. It allows you to hide certain parts of the post which the reader can expand by pressing the summary of the hidden text. Wanna try it?

2 Likes

Thanks, I was a little worried about taking up a huge block of space in a single reply. Looks a bit nicer now!

1 Like

This is an interesting solution, though I’ll admit it’s a little involved for an average user (though then again, an average user might not care about the functionality i’m after). The main turn-off for me is the fact that it doesn’t scale well (every tweak to my backup job becomes 3x more complicated to update), and seemingly loses the benefits of going through the GUI (statistics, metrics, logs(?) wouldn’t reflect reality it sounds like).

Do you have reason to believe that each run would create a new version? Or that the database bogs down in some specific way after a lot of versions are backed up?

I should write a small commandline tool that sends the commands to the server, such that you can start/stop jobs from the commandline. There is a python script that serves as an example of how to do it:

2 Likes

I do because files change all the time so its pretty much a given that hourly run will generate a new version every time. And yes since common sense and years of working with various databases suggest that the more data there is in the database, the slower it is. But my question was more about how well does Duplicati handle a large number of versions. For example, is there code to limit selects to a reasonable (retrieval time wise) amount of versions and paginate results?

Just to throw a number out there: running hourly backups with a 6 months retention policy will generate ~4320 versions.

No, not at this time. This is slightly mitigated by the limitation that only a single version can be displayed for restoring (in other words, the problem will appear once multiple versions can be browsed).

1 Like

@drakar2007, will a “thread-priority” Advanced Option setting of “Idle” do close to what you want?

From my limited knowledge of thread priority options, I assume that such an option would cause the Duplicati engine take lower priority than any other tasks. It seems like this would have some overlap in functionality and use cases, but of course not total overlap - for instance I figure there could be a case where I want to manually start a backup job and have it take normal priority until finishing, whether or not I’m using the system, etc.

But yes, I think such an option would definitely be an improvement, especially if there is some flexibility in the configuration given.

@drakar2007, that’s a good point - there are times when I could see wanting to go “full speed ahead”. It might be nice to add a “CPU icon” next to the play/pause and bandwidth throttling icons allowing for manual priority setting of the currently running job.

I’m pretty sure in Windows a thread can dynamically have it’s priority adjusted so I assume the same can be done in Linux and Mac OS.

1 Like

thread-priority is a useful feature, but for my case it’s not what I want. In my case I would actually prefer that Duplicati uses as much resources as it needs to finish quickly and then let my computer go back to being idle. So I really just want duplicati to be able to delay starting a backup until my computer is idle.

I’m going to try u/TPSMono instructions for using task scheduler, thanks TPSMono for writing up a guide!

Btw, there is a feature request for this, but it is from 2014 and I have no idea if it is being worked on: Postpone the backup until computer is idle · Issue #520 · duplicati/duplicati · GitHub