Running script before with arguments on duplicati-2.1.0.3-beta.x86_64

[Louis_van_Dyk]

I see a number of tickets have been logged before about running a script before or after a job, and especially asking about command line arguments. There is even an example file included in the duplicati package called: run-script-example.sh

Here we are told that we can check the boolean box for --run-script-with-arguments and that we should quote arguments. This then offers a most confusing example with double quotes, backslash quoting, and an unexpected number of double-quotes. Trying this in a live job just doesn’t work.

Can you please assist me to do the following:
Before the job, run the script
/path/to/script/scriptname.sh -a arg1 -b arg2 anotherparameter
and then after the job to do the same with other arguments.

If I use just one script that runs the command that I want, I get the line
[Information-Duplicati.Library.Main.Controller-AbortOperation]: Aborting operation by request, requested result: Normal
and the backup ends as “Interrupted”.

If I use the run script before required, then it fails with error “returned with exit code 1”.

What I’m trying to do is take a snapshot of a virtual machine before I back it up, and after the backup release the snapshot. This allows a consistent image to be backed up, making for a more reliable restore point.

[ts678]

@Louis_van_Dyk

There is even an example file included in the duplicati package called: run-script-example.sh

Please read near the top:

# --run-script-before = <filename>
# Duplicati will run the script before the backup job and waits for its 
# completion for 60 seconds (default timeout value). After a timeout a 
# warning is logged and the backup is started.
# The following exit codes are supported:
#
# - 0: OK, run operation
# - 1: OK, don't run operation
# - 2: Warning, run operation
# - 3: Warning, don't run operation
# - 4: Error, run operation
# - 5: Error don't run operation
# - other: Error don't run operation

@Louis_van_Dyk

If I use the run script before required, then it fails with error “returned with exit code 1”.

--run-script-before-required (Path): Run a required script on startup Execute a script before performing an operation. The operation will block until the script has completed or timed out. If the script returns a non-zero error code or times out, the operation will be aborted.

This predates the fancier handling in the example, and you can decide which plan you like.

@Louis_van_Dyk

I get the line

Where? Is this a GUI setup, and you’re in Complete log in the job log? Exit code 1 does:

  "Messages": [
    "2025-02-08 10:58:04 -05 - [Information-Duplicati.Library.Main.Controller-AbortOperation]: Aborting operation by request, requested result: Normal"
  ],

for reasons explained at top. Exit code 1 means OK, don't run operation (as reported).

@Louis_van_Dyk

This then offers a most confusing example with double quotes, backslash quoting, and an unexpected number of double-quotes.

Have you used shell and used quotes around a word sequence that must be kept together?
Bash takes either double or single quotes for that. Windows Command Prompt uses double.
The thing with double quotes is if you want literal double quote inside, you backslash before.

If I’m correctly understanding the the example, I think it’s for CLI, where quotes matter more.
If you’re in the GUI, you don’t have a shell to interpret, but you do have Duplicati and similar technique to group word sequences. If you have no embedded spaces, you need no quotes.

An example in bash of how to quote (and not quote):

$ echo -->"<--
> ^C
$ echo '-->"<--'
-->"<--
$ echo -->"<--
> ^C
$ echo "-->"<--"
> ^C
$ echo "-->\"<--"
-->"<--
$ 

EDIT:

/path/to/script/scriptname.sh -a arg1 -b arg2 anotherparameter

I don’t know if there are any embedded spaces in that. If there are, then quote as documented:

--run-script-with-arguments (Boolean): Enable script arguments This option enables the use of script arguments. If this option is set, the script arguments are treated as commandline strings. Use single or double quotes to separate arguments.

I’m not sure I like the “separate arguments” phrasing. I’d probably view it as “group arguments”, because just a space will separate arguments – except sometimes you don’t want separation…

[Louis_van_Dyk]

Hi, I would say I am well versed in Linux and understand quoting. It was late, and I could have stated by case better, I suppose. So let me clarify …

(TLDR, I’ve solved it, but the documentation needs tweaking.)

I’m using the GUI to set up. In it, I’ve added run-script-before-required and also a run-script-after. (I understand the exit codes and their implication. The script however, runs from the CLI with an exit code of 0.) I’ve also added the run-script-timeout option to 60 minutes, and then checked the box on run-script-with-arguments.

The description on run-script-with-arguments states the following:

This option enables the use of script arguments. If this option is set, the script arguments are treated as commandline strings. Use single or double quotes to separate arguments.

Default value: “false”

Lastly, when I look at the example script, /usr/lib/duplicati/run-script-example.sh, it states the following:

# --run-script-with-arguments = <boolean>
# If set to true, the script path will be parsed as a command line, and the
# arguments will be passed to the script. If set to false (default), 
# the script path will used as a single path.
# Use double- or single-quotes to pass arguments with spaces.
# Example: --run-script-before="\"/path to/script.sh\" \"arg1 \" arg2"

The last line now is the confusing example to which I’m referring.

"\"/path to/script.sh\" \"arg1 \" arg2"

I’ve managed to fix the fault so that the script runs successfully. For those making the journey, let me elaborate on what I had to do:

I am running
https://raw.githubusercontent.com/dguerri/LibVirtKvm-scripts/master/fi-backup.sh

So in my run-script-before-required field, I enter
/usr/local/bin/fi-backup -q -v all

If I run the job, in the log for the job after it runs, I get

2025-02-09 01:03:34 +02 - [Error-Duplicati.Library.Main.Controller-FailedOperation]: The operation Backup has failed with error: The script “/usr/local/bin/fi-backup.sh” returned with exit code 1
UserInformationException: The script “/usr/local/bin/fi-backup.sh” returned with exit code 1

I decided to run the command from the command line, and then tested the exit code … sigh … facepalm.

# echo $?
1

I have multiple Virtual Machines, but not all of them are running. If I ran it against the only VM that’s running (and is normally always on), I got:

# echo $?
0

So with that hurdle crossed, I had to figure out how the quoting process actually should be. I must note here that the documentation in the /usr/lib/duplicati/run-script-example.sh file needs to be fixed. It is atrocious.

I got the job to run successully when I used the line:
"/usr/local/bin/fi-backup.sh" "-q" "-v" "MyRunningVM"

Thank you for bearing with me an being an ear to bounce things off of.

[ts678]

@Louis_van_Dyk

I got the job to run successully when I used the line:
"/usr/local/bin/fi-backup.sh" "-q" "-v" "MyRunningVM"

I’m thinking essential change there was use of MyRunningVM. If no spaces, no quotes needed.

I’m not sure you’re following what I mean by CLI versus GUI quoting, so here’s what bash does:

$ echo "\"/path to/script.sh\" \"arg1 \" arg2"
"/path to/script.sh" "arg1 " arg2
$ 

Script path has an embedded space, arg1 for some reason has a trailing space, arg2 has none.

Without quotes, bash would think it was told to run /path, as below. Duplicati would hit this too:

$ /path to/script.sh
bash: /path: No such file or directory
$ 

I was testing with a script logargs.sh:

$ cat logargs.sh
#!/bin/bash
IFS=$'\n'
echo "$*" > /tmp/logargs.log
exit 0
$ 

and thinking that your case works fine (as expected) without the double quotes. For example

/tmp/logargs.sh -q -v all

gets

$ cat logargs.log
-q
-v
all
$ 

and

"/tmp/logargs.sh" "-q" "-v" "MyRunningVM"

gets

$ cat logargs.log
-q
-v
MyRunningVM
$ 

@Louis_van_Dyk

It is atrocious.

It could certainly say more, i.e. be an explained example rather than one reader must interpret.
I’m not sure I’m against double-quote escaping for use from a CLI, as all of this is CLI-oriented.
That use could be explained. For bash (but not Windows) single quotes would be easier to do.
Put single quotes around the whole string (assuming it has none in it) and don’t escape double.

In below testing, sayargs.sh is like logargs.sh except going to stdout instead of the logargs.log:

$ ./sayargs.sh '"/path to/script.sh" "arg1word1 arg1word2" arg2'
"/path to/script.sh" "arg1word1 arg1word2" arg2
$ 

A more realistic example might have option switches, as quotes keep the option value together.
Show a GUI example equivalent to the CLI example. It should be simpler without extra quoting.
This might go first, as it gets right to the Duplicati part of it. After that, explain dealing with shell.

Was this the main objection or is there more? The developer is usually willing to improve things.

[Louis_van_Dyk]

Oh my goodness! You are right. Once I didn’t use the “all” keyword, but rather just the “MyRunningVM”, I can run the command without any quotes! So this was rather a case of my missing that the error 1 was actually as a result of the script, and not Duplicati’s interpretation. I think I was thrown by the error message only showing the script in the error and not including the arguments. (I thought it was failing because it wasn’t passing the arguments to the script, as opposed to how it was actually failing because it was reading them, and that when it ran it in fact generated the error).

On the documentation in the run-script-example.sh I think the fact that arg1 had a trailing space and was in quotes, while arg2 didn’t have quotes, was what was confusing me. I do understand that a path with a space must be quoted. And from the description text in the GUI though ALL arguments must be quoted, but wasn’t sure if they had to be individually quoted (i.e. 3 independently quoted items) or all arguments surrounded by one quote (i.e. arguments passed as one long string).

I think in the end, what I’m hoping can be achieved is just that the documentation supporting the arguments in the example script can be less convoluted. (refer trailing space – who does that??)

I was also thrown by comments on older threads that said that arguments were not supported yet, and only in canary. That article was less than 6 months ago.
https://forum.duplicati.com/t/run-script-with-parameters-error-in-log/19198
… so that just added to my confusion.

Once again thanks for the help. I think we can close this one now.

[ts678]

@Louis_van_Dyk

That article was less than 6 months ago.

Things sometimes change. The Canary announcement was v2.0.9.101_canary_2024-06-27.

Added support for arguments for scripts

Although it’s hard recently to know what got in a Beta, I think a later Canary release became:

v2.1.0.2_beta_2024-11-29 but About → Changelog never mentioned this again, yet it’s there.

Maybe some documentation upgrades can be done before other people notice the addition…

[kenkendk]

@Louis_van_Dyk

I think in the end, what I’m hoping can be achieved is just that the documentation supporting the arguments in the example script can be less convoluted. (refer trailing space – who does that??)

If you have a few examples, I can easily add them.
And yes, trailing space is really annoying and should be avoided, but I also wanted the example to cover something like that.

@ts678

Maybe some documentation upgrades can be done before other people notice the addition…

Is the confusion here that not all arguments need to be quoted? Or that you are dealing with both shell-quotes and option quotes?

[ts678]

@kenkendk

Is the confusion here that not all arguments need to be quoted? Or that you are dealing with both shell-quotes and option quotes?

Both, detailed already, and user might have a different opinion, so feel free to comment as well.

Without repeating everything I wrote, it’s an odd example, unexplained, left to user to figure out.

Positional arguments are unusual, so maybe you can show an option taking string with spaces.

Starting with Duplicati-level basics in GUI form (looking only at its quote plan) avoids overloads.

After that (if need be) adding the further oddness of quoting to protect from shell may be helpful.

I think Command Prompt is stuck with double quotes. I’m not sure what quoting PowerShell has.

I’m thinking that something elegant like font changes are probably hard, as plain text help exists.

[Louis_van_Dyk]

@kenkendk

Is the confusion here that not all arguments need to be quoted? Or that you are dealing with both shell-quotes and option quotes?

For me, there were a couple of things:

1. Based on threads on the support channels, I wasn’t sure if the arguments were working, which it now seems they are … so thank you for that.
2. The GUI info screen said to quote arguments. This was a little vague, as I ended up quoting EVERY argument individually, as well as the path and script name (even though I had no spaces in that).
3. The example script with that rather nasty trailing space had me thrown off becasue I couldn’t understand why arg2 didn’t have quotes around it - because I was under the assumption that EVERYTHING had to be quoted.

I’d say if the info text could just say that “items with spaces need to be quoted with double or single quotes”, that that would solve a whole lot of confusion.

Thank you for your efforts - they are greatly appreciated.

[kenkendk]

@Louis_van_Dyk

I’d say if the info text could just say that “items with spaces need to be quoted with double or single quotes”, that that would solve a whole lot of confusion.

I have made a suggestion to new text here:

Improve shell-script arguments

[Louis_van_Dyk]

@kenkendk

I have made a suggestion to new text here

Perfect! Future Duplicati Users salute you!