Hi All,
I wanted to give you a watchdog script for Win11 I’m using for Duplicati.
The description is in the code header.
Have fun!
<#
.SYNOPSIS
Duplicati Watchdog for Windows PowerShell 7.
.DESCRIPTION
This script provides an independent watchdog for Duplicati backup jobs.
The script is designed to be used with Duplicati's "run-script-before"
mechanism.
When Duplicati starts a backup, it sets the environment variable:
DUPLICATI__backup_name
The script immediately starts a second, hidden PowerShell process and
returns to Duplicati. The hidden process then monitors ONLY the task
belonging to the backup job that started it.
The watchdog is therefore independent from the Duplicati backup process
itself and does not depend on "run-script-after".
Normal operation:
1. Duplicati starts the script.
2. The script starts an independent hidden watchdog process.
3. The original script exits immediately.
4. The watchdog authenticates against the local Duplicati API.
5. It resolves the backup name to the BackupID.
6. It identifies the corresponding TaskID.
7. It monitors that exact TaskID.
8. If the task disappears normally, the watchdog exits immediately.
Timeout handling:
After the configured timeout has elapsed while the task is still
active, the watchdog escalates as follows:
1. POST /api/v1/task/<TaskID>/stop
2. Wait StopWaitSeconds
3. Check the SAME TaskID again
4. If still active:
POST /api/v1/task/<TaskID>/abort
5. Wait AbortWaitSeconds
6. Check the SAME TaskID again
7. If still active:
terminate Duplicati.Server.exe
8. Wait ServerRestartWaitSeconds
9. Restart Duplicati.Server.exe if it has not already restarted.
The watchdog never kills a task merely because another Duplicati task
is running. Every watchdog instance is tied to its own TaskID.
.SECURITY
The Duplicati server password is NOT stored in this script.
The script reads the password from Windows Credential Manager.
Configure the credential once with Duplicati SecretTool, for example:
C:\Program Files\Duplicati 2\Duplicati.CommandLine.SecretTool.exe `
set "wincred://" `
"duplicati-serverutil-password" `
"YOUR-DUPLICATI-SERVER-PASSWORD"
The credential name used by this script is:
duplicati-serverutil-password
The password is read directly from Windows Credential Manager using
CredRead(). SecretTool's "get" command is NOT required.
The script only communicates with the local Duplicati server:
http://127.0.0.1:8200
Duplicati's normal API access uses a short-lived access token. The
watchdog obtains a token when required and uses it as:
Authorization: Bearer <token>
.REQUIREMENTS
- Windows
- PowerShell 7.x
- Duplicati 2.x
- Duplicati.Server.exe running
- Duplicati server API available on localhost
- A Duplicati server password stored in Windows Credential Manager
- The account running this script must be able to read that credential
.INSTALLATION
1. Save this file as:
C:\Users\<username>\bin\Duplicati\DuplicatiWatchdog.ps1
The script itself does not contain a hard-coded Windows username.
2. Store the Duplicati server password in Windows Credential Manager
using Duplicati SecretTool:
C:\Program Files\Duplicati 2\Duplicati.CommandLine.SecretTool.exe `
set "wincred://" `
"duplicati-serverutil-password" `
"YOUR-DUPLICATI-SERVER-PASSWORD"
3. Configure Duplicati's "run-script-before" command.
Example:
"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" `
-ExecutionPolicy Bypass `
-File "C:\Users\<username>\bin\Duplicati\DuplicatiWatchdog.ps1"
Replace <username> with the Windows account used by Duplicati.
Alternatively, when configuring the path programmatically, the
PowerShell environment variable $env:USERNAME can be used to build
the path.
.CONFIGURATION
The following values can be changed in the CONFIGURATION section:
TimeoutMinutes
Maximum time the backup task may remain active before escalation.
PollIntervalSeconds
Interval between task status checks.
StopWaitSeconds
Time to wait after the stop request.
AbortWaitSeconds
Time to wait after the abort request.
ServerRestartWaitSeconds
Time to wait after killing Duplicati.Server.exe before checking
whether it has already restarted.
TestMode
If $true, timeout escalation is simulated but no stop, abort or
server termination is performed.
DuplicatiServerUrl
Local Duplicati API URL.
ServerExe
Path to Duplicati.Server.exe.
CredentialName
Windows Credential Manager target name.
.NOTES
The watchdog intentionally does not create a separate log file.
Output from the independent watchdog process cannot reliably be attached
to the original Duplicati job output.
The script is intentionally self-contained and has no external PowerShell
module dependencies.
.VERSION
1.0.0
#>
param(
[switch]$Watchdog
)
# ============================================================================
# CONFIGURATION
# ============================================================================
$TimeoutMinutes = 10
$PollIntervalSeconds = 5
$StopWaitSeconds = 10
$AbortWaitSeconds = 10
$ServerRestartWaitSeconds = 15
$TestMode = $false
$DuplicatiServerUrl = 'http://127.0.0.1:8200'
$ServerExe = 'C:\Program Files\Duplicati 2\Duplicati.Server.exe'
$CredentialName = 'duplicati-serverutil-password'
# ============================================================================
# START INDEPENDENT WATCHDOG
# ============================================================================
if (-not $Watchdog) {
$PowerShellExe = Join-Path $PSHOME 'powershell.exe'
if (-not (Test-Path -LiteralPath $PowerShellExe)) {
exit 0
}
# $PSCommandPath always points to THIS script.
#
# No username is hard-coded here.
#
# This also makes the script portable if the user's profile directory
# changes.
$ArgumentList = @(
'-NoProfile'
'-ExecutionPolicy'
'Bypass'
'-File'
"`"$PSCommandPath`""
'-Watchdog'
)
try {
Start-Process `
-FilePath $PowerShellExe `
-ArgumentList $ArgumentList `
-WindowStyle Hidden `
-ErrorAction Stop | Out-Null
}
catch {
# The watchdog must never make the Duplicati "before" script fail.
}
exit 0
}
# ============================================================================
# WATCHDOG PROCESS
# ============================================================================
$ErrorActionPreference = 'Stop'
$BackupName = $env:DUPLICATI__backup_name
if ([string]::IsNullOrWhiteSpace($BackupName)) {
exit 0
}
# ============================================================================
# WINDOWS CREDENTIAL MANAGER ACCESS
# ============================================================================
if (-not ('DuplicatiWatchdogCredential' -as [type])) {
Add-Type @'
using System;
using System.Runtime.InteropServices;
public static class DuplicatiWatchdogCredential
{
[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Unicode)]
private struct CREDENTIAL
{
public int Flags;
public int Type;
public IntPtr TargetName;
public IntPtr Comment;
public System.Runtime.InteropServices.ComTypes.FILETIME LastWritten;
public int CredentialBlobSize;
public IntPtr CredentialBlob;
public int Persist;
public int AttributeCount;
public IntPtr Attributes;
public IntPtr TargetAlias;
public IntPtr UserName;
}
[DllImport("advapi32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
private static extern bool CredRead(
string target,
int type,
int flags,
out IntPtr credential
);
[DllImport("advapi32.dll", SetLastError = true)]
private static extern void CredFree(IntPtr credential);
public static string Read(string target)
{
IntPtr credentialPtr;
if (!CredRead(target, 1, 0, out credentialPtr))
{
int error = Marshal.GetLastWin32Error();
throw new InvalidOperationException(
"Windows Credential Manager could not read credential '" +
target + "'. Win32 error: " + error
);
}
try
{
CREDENTIAL credential =
Marshal.PtrToStructure<CREDENTIAL>(credentialPtr);
if (credential.CredentialBlob == IntPtr.Zero ||
credential.CredentialBlobSize <= 0)
{
return String.Empty;
}
return Marshal.PtrToStringUni(
credential.CredentialBlob,
credential.CredentialBlobSize / 2
);
}
finally
{
CredFree(credentialPtr);
}
}
}
'@
}
# ============================================================================
# READ Duplicati SERVER PASSWORD
# ============================================================================
try {
$ServerPassword =
[DuplicatiWatchdogCredential]::Read($CredentialName)
}
catch {
exit 0
}
if ([string]::IsNullOrWhiteSpace($ServerPassword)) {
exit 0
}
# ============================================================================
# API AUTHENTICATION
# ============================================================================
$AccessToken = $null
function Get-DuplicatiAccessToken {
$loginUri = "$DuplicatiServerUrl/api/v1/login"
# Duplicati's current API uses the server password to obtain an
# access token. There is no username.
$body = @{
Password = $ServerPassword
} | ConvertTo-Json -Compress
try {
$response = Invoke-RestMethod `
-Uri $loginUri `
-Method Post `
-ContentType 'application/json' `
-Body $body `
-TimeoutSec 30 `
-ErrorAction Stop
}
catch {
throw "Duplicati API login failed: $($_.Exception.Message)"
}
if ($null -eq $response) {
throw 'Duplicati API login returned an empty response.'
}
if ($response.AccessToken) {
return [string]$response.AccessToken
}
if ($response.access_token) {
return [string]$response.access_token
}
if ($response.Token) {
return [string]$response.Token
}
if ($response.token) {
return [string]$response.token
}
throw 'Duplicati API login succeeded but no access token was returned.'
}
# ============================================================================
# API REQUEST HELPERS
# ============================================================================
function Invoke-DuplicatiGet {
param(
[Parameter(Mandatory)]
[string]$Path
)
if ([string]::IsNullOrWhiteSpace($AccessToken)) {
$script:AccessToken = Get-DuplicatiAccessToken
}
$uri = "$DuplicatiServerUrl$Path"
try {
return Invoke-RestMethod `
-Uri $uri `
-Method Get `
-Headers @{
Authorization = "Bearer $AccessToken"
} `
-TimeoutSec 30 `
-ErrorAction Stop
}
catch {
# Access tokens are short-lived. Re-authenticate once and retry.
try {
$script:AccessToken = Get-DuplicatiAccessToken
return Invoke-RestMethod `
-Uri $uri `
-Method Get `
-Headers @{
Authorization = "Bearer $AccessToken"
} `
-TimeoutSec 30 `
-ErrorAction Stop
}
catch {
throw
}
}
}
function Invoke-DuplicatiPost {
param(
[Parameter(Mandatory)]
[string]$Path
)
if ([string]::IsNullOrWhiteSpace($AccessToken)) {
$script:AccessToken = Get-DuplicatiAccessToken
}
$uri = "$DuplicatiServerUrl$Path"
try {
return Invoke-RestMethod `
-Uri $uri `
-Method Post `
-Headers @{
Authorization = "Bearer $AccessToken"
} `
-TimeoutSec 30 `
-ErrorAction Stop
}
catch {
# Re-authenticate once in case the access token expired.
try {
$script:AccessToken = Get-DuplicatiAccessToken
return Invoke-RestMethod `
-Uri $uri `
-Method Post `
-Headers @{
Authorization = "Bearer $AccessToken"
} `
-TimeoutSec 30 `
-ErrorAction Stop
}
catch {
throw
}
}
}
# ============================================================================
# RESOLVE BACKUP NAME -> BACKUP ID
# ============================================================================
try {
$BackupsResponse = Invoke-DuplicatiGet '/api/v1/backups'
}
catch {
exit 0
}
$Backups = $BackupsResponse
# Some API versions wrap the result in a property.
if ($BackupsResponse.Backups) {
$Backups = $BackupsResponse.Backups
}
elseif ($BackupsResponse.Data) {
$Backups = $BackupsResponse.Data
}
$BackupId = $null
foreach ($Backup in @($Backups)) {
$Name = $null
$Id = $null
if ($Backup.Name) {
$Name = [string]$Backup.Name
}
if ($Backup.ID) {
$Id = [string]$Backup.ID
}
elseif ($Backup.Id) {
$Id = [string]$Backup.Id
}
elseif ($Backup.BackupID) {
$Id = [string]$Backup.BackupID
}
if (
-not [string]::IsNullOrWhiteSpace($Name) -and
$Name -eq $BackupName
) {
$BackupId = $Id
break
}
}
if ([string]::IsNullOrWhiteSpace($BackupId)) {
exit 0
}
# ============================================================================
# FIND THE TASK BELONGING TO THIS BACKUP
# ============================================================================
$TaskId = $null
function Get-OwnTask {
param(
[Parameter(Mandatory)]
[string]$ExpectedBackupId
)
try {
$TasksResponse = Invoke-DuplicatiGet '/api/v1/tasks'
}
catch {
return $null
}
$Tasks = $TasksResponse
if ($TasksResponse.Tasks) {
$Tasks = $TasksResponse.Tasks
}
elseif ($TasksResponse.Data) {
$Tasks = $TasksResponse.Data
}
foreach ($Task in @($Tasks)) {
$TaskBackupId = $null
$CurrentTaskId = $null
if ($Task.BackupId) {
$TaskBackupId = [string]$Task.BackupId
}
elseif ($Task.BackupID) {
$TaskBackupId = [string]$Task.BackupID
}
elseif ($Task.Backup) {
if ($Task.Backup.ID) {
$TaskBackupId = [string]$Task.Backup.ID
}
elseif ($Task.Backup.Id) {
$TaskBackupId = [string]$Task.Backup.Id
}
}
if ($Task.Task) {
$CurrentTaskId = [string]$Task.Task
}
elseif ($Task.TaskId) {
$CurrentTaskId = [string]$Task.TaskId
}
elseif ($Task.TaskID) {
$CurrentTaskId = [string]$Task.TaskID
}
elseif ($Task.ID) {
$CurrentTaskId = [string]$Task.ID
}
elseif ($Task.Id) {
$CurrentTaskId = [string]$Task.Id
}
if (
-not [string]::IsNullOrWhiteSpace($CurrentTaskId) -and
$TaskBackupId -eq $ExpectedBackupId
) {
return $CurrentTaskId
}
}
return $null
}
# ============================================================================
# WAIT FOR THIS BACKUP'S TASK
# ============================================================================
while ($null -eq $TaskId) {
$TaskId = Get-OwnTask -ExpectedBackupId $BackupId
if ($null -eq $TaskId) {
Start-Sleep -Seconds $PollIntervalSeconds
}
}
# ============================================================================
# CHECK WHETHER THIS EXACT TASK IS STILL ACTIVE
# ============================================================================
function Test-OwnTaskActive {
param(
[Parameter(Mandatory)]
[string]$ExpectedTaskId,
[Parameter(Mandatory)]
[string]$ExpectedBackupId
)
try {
$TasksResponse = Invoke-DuplicatiGet '/api/v1/tasks'
}
catch {
# A temporary API error must NOT cause the watchdog to kill the
# Duplicati server. Treat the task as active and try again later.
return $true
}
$Tasks = $TasksResponse
if ($TasksResponse.Tasks) {
$Tasks = $TasksResponse.Tasks
}
elseif ($TasksResponse.Data) {
$Tasks = $TasksResponse.Data
}
foreach ($Task in @($Tasks)) {
$CurrentTaskId = $null
$CurrentBackupId = $null
if ($Task.Task) {
$CurrentTaskId = [string]$Task.Task
}
elseif ($Task.TaskId) {
$CurrentTaskId = [string]$Task.TaskId
}
elseif ($Task.TaskID) {
$CurrentTaskId = [string]$Task.TaskID
}
elseif ($Task.ID) {
$CurrentTaskId = [string]$Task.ID
}
elseif ($Task.Id) {
$CurrentTaskId = [string]$Task.Id
}
if ($Task.BackupId) {
$CurrentBackupId = [string]$Task.BackupId
}
elseif ($Task.BackupID) {
$CurrentBackupId = [string]$Task.BackupID
}
elseif ($Task.Backup) {
if ($Task.Backup.ID) {
$CurrentBackupId = [string]$Task.Backup.ID
}
elseif ($Task.Backup.Id) {
$CurrentBackupId = [string]$Task.Backup.Id
}
}
if (
$CurrentTaskId -eq $ExpectedTaskId -and
$CurrentBackupId -eq $ExpectedBackupId
) {
return $true
}
}
# The exact task is no longer present.
#
# This is the normal completion path.
return $false
}
# ============================================================================
# MONITOR TASK UNTIL TIMEOUT OR NORMAL COMPLETION
# ============================================================================
$Deadline = (Get-Date).AddMinutes($TimeoutMinutes)
while ((Get-Date) -lt $Deadline) {
if (-not (Test-OwnTaskActive `
-ExpectedTaskId $TaskId `
-ExpectedBackupId $BackupId)) {
exit 0
}
Start-Sleep -Seconds $PollIntervalSeconds
}
# ============================================================================
# FINAL CHECK BEFORE ESCALATION
# ============================================================================
if (-not (Test-OwnTaskActive `
-ExpectedTaskId $TaskId `
-ExpectedBackupId $BackupId)) {
exit 0
}
# ============================================================================
# TEST MODE
# ============================================================================
if ($TestMode) {
# In TestMode no destructive action is performed.
exit 0
}
# ============================================================================
# ESCALATION LEVEL 1: STOP
# ============================================================================
try {
Invoke-DuplicatiPost "/api/v1/task/$TaskId/stop" | Out-Null
}
catch {
# Continue with the verification step.
}
Start-Sleep -Seconds $StopWaitSeconds
if (-not (Test-OwnTaskActive `
-ExpectedTaskId $TaskId `
-ExpectedBackupId $BackupId)) {
exit 0
}
# ============================================================================
# ESCALATION LEVEL 2: ABORT
# ============================================================================
try {
Invoke-DuplicatiPost "/api/v1/task/$TaskId/abort" | Out-Null
}
catch {
# Continue with the verification step.
}
Start-Sleep -Seconds $AbortWaitSeconds
if (-not (Test-OwnTaskActive `
-ExpectedTaskId $TaskId `
-ExpectedBackupId $BackupId)) {
exit 0
}
# ============================================================================
# ESCALATION LEVEL 3: TERMINATE Duplicati.Server.exe
# ============================================================================
$ServerProcesses = @()
try {
$ServerProcesses = @(
Get-CimInstance Win32_Process `
-Filter "Name = 'Duplicati.Server.exe'" `
-ErrorAction Stop
)
}
catch {
exit 0
}
if ($ServerProcesses.Count -eq 0) {
exit 0
}
# We normally expect one server process.
#
# If several processes exist, terminate only the instances found here.
# The original command line and executable path are retained so the process
# can be restarted using the same executable and arguments.
$RestartCandidates = @()
foreach ($Process in $ServerProcesses) {
$RestartCandidates += [PSCustomObject]@{
ProcessId = [int]$Process.ProcessId
CommandLine = [string]$Process.CommandLine
ExecutablePath = [string]$Process.ExecutablePath
}
}
# ============================================================================
# LAST TASK CHECK BEFORE KILLING SERVER
# ============================================================================
if (-not (Test-OwnTaskActive `
-ExpectedTaskId $TaskId `
-ExpectedBackupId $BackupId)) {
exit 0
}
# ============================================================================
# TERMINATE SERVER
# ============================================================================
foreach ($Candidate in $RestartCandidates) {
try {
Stop-Process `
-Id $Candidate.ProcessId `
-Force `
-ErrorAction SilentlyContinue
}
catch {
# Continue with the other server process, if any.
}
}
# ============================================================================
# WAIT FOR SERVER TO RESTART
# ============================================================================
Start-Sleep -Seconds $ServerRestartWaitSeconds
# ============================================================================
# CHECK WHETHER Duplicati.Server.exe ALREADY RESTARTED
# ============================================================================
$RunningServer = @()
try {
$RunningServer = @(
Get-CimInstance Win32_Process `
-Filter "Name = 'Duplicati.Server.exe'" `
-ErrorAction SilentlyContinue
)
}
catch {
$RunningServer = @()
}
if ($RunningServer.Count -gt 0) {
exit 0
}
# ============================================================================
# RESTART Duplicati.Server.exe
# ============================================================================
foreach ($Candidate in $RestartCandidates) {
$ExecutablePath = $Candidate.ExecutablePath
$CommandLine = $Candidate.CommandLine
if ([string]::IsNullOrWhiteSpace($ExecutablePath)) {
$ExecutablePath = $ServerExe
}
if (-not (Test-Path -LiteralPath $ExecutablePath)) {
continue
}
# Try to reconstruct the original command line.
#
# The executable itself is removed from the beginning. The remaining
# command line is passed as a single argument string to Start-Process.
$Arguments = $null
if (-not [string]::IsNullOrWhiteSpace($CommandLine)) {
$QuotedExecutable =
'"' + $ExecutablePath + '"'
if ($CommandLine.StartsWith($QuotedExecutable,
[System.StringComparison]::OrdinalIgnoreCase)) {
$Arguments = $CommandLine.Substring(
$QuotedExecutable.Length
).Trim()
}
elseif ($CommandLine.StartsWith($ExecutablePath,
[System.StringComparison]::OrdinalIgnoreCase)) {
$Arguments = $CommandLine.Substring(
$ExecutablePath.Length
).Trim()
}
}
try {
if ([string]::IsNullOrWhiteSpace($Arguments)) {
Start-Process `
-FilePath $ExecutablePath `
-WindowStyle Hidden `
-ErrorAction Stop | Out-Null
}
else {
Start-Process `
-FilePath $ExecutablePath `
-ArgumentList $Arguments `
-WindowStyle Hidden `
-ErrorAction Stop | Out-Null
}
break
}
catch {
# Try the next candidate, if there is one.
}
}
exit 0