The command “duplicati-cli --parameters-file=”/tmp/tmp3a22dobf" list"` works as expected.
The command “duplicati-cli --parameters-file=”/tmp/tmp3a22dobf" list does-not-exist" should look for the file “does-not-exist” in backups, but instead lists filesets.
The command “duplicati-cli --parameters-file=”/tmp/tmp3a22dobf" list does-not-exist does-not-exist-too" looks for something in the backups:
duplicati-cli --parameters-file="/tmp/tmp3a22dobf" list does-not-exist does-not-exist-too
No files matching, looking in all versions
No files matched expression
The command “duplicati-cli --parameters-file=”/tmp/tmp3a22dobf" list ‘’ does-not-exist" (note the null parameter!) more-or-less works as well:
duplicati-cli --parameters-file="/tmp/tmp3a22dobf" list '' does-not-exist
No files matching, looking in all versions
No files matched expression
I’m guessing that the “list” command is looking for “storage-URL” as per the help output, but the URL is provided in the parameters-file. Order of parameters on the command-line don’t seem to matter.
Thanks for reporting this with an easy to follow sample.
I have located the problem, and it is replacing the “existing url”.
In other words, it expects:
duplicati-cli --parameters-file=”/tmp/tmp3a22dobf" list dummy://abc does-not-exist
And then will replace dummy://abc with the real target.
In your case, there is no target, so it replaces whatever is at that position (does-not-exist):
duplicati-cli --parameters-file=”/tmp/tmp3a22dobf" list does-not-exist
I can see why this is confusing, but I am not sure how we should handle it.
We don’t know if the input contains a target already or some other parameter at that position.
The easy path would be to clearly document that Target-URL on the command line is required - but ignored - if provided in a parameters-file.
A “breaking” change would be to make any parameter that can be specified as an “option” in a parameters-file an “option” on the command line as well. I.e., instead of a positional parameter Target-URL would have to be specified on the command line as --target=Target-URL.
Another “breaking” change would be to parse the command line once for positional parameters, and then a second time for options, throwing an error if Target-URL was never provided. This depends on how you are doing the parsing; if using a standard library of some sort this may not be easily doable (I’ve never programmed in C# and the last time I wrote a program for Microsoft Windows it used Win32, so must please ignorance of how easy this would be).
Living in a Linux world (and Solaris before that and VMS before that), I tend to favor CLI interfaces. After resolving my recent issues, I was looking for a cli method to provide the equivalent of the main GUI page - listing backups, last known good backup, schedule, etc. I have more-or-less learned the various schemas and relations in the Duplicati-server.sqlite database, and this in turn made me realize I could use this data to provide the status info, and also to address one of my annoyances - specifying backup-related parameters when issuing duplicati-cli commands. I created a python script that lets me specify the backup name as a required option, and the arguments for the duplicati-cli command - other than the Target-URL, db path, and passphrase which it extracts from the server database. For security, it puts those parameters in a parameters-file. Since I put the Target-URL in the file, I was not expecting to have to provide it as a command-line argument, and hence ran into the “list” behavior I reported,
So basically I can do something like:
sudo duplicati-helper list [--backup name-regex]
sudo duplicati-helper cli --backup backup-name list [filename [...]]
I can certainly allow for the “ignored” Target-URL parameter in my script, but wanted to report the behavior in case it wasn’t intentional.
I have been thinking about this some more, and while I don’t have any new suggestions, here are some thoughts on design, etc.
The basic syntax for the cli is “program action object action-parameters”. “program” is duplicati-cli (at least on Linux). “action” is backup, restore, list, etc. “object” is the thing that “action” will take action on.
For most actions, “object” is a backup, which means that “object” needs to identify which backup. Technically, the backup is identified by a URI - anything local is effectively a cache. Therefore the URI has to be a required parameter for any “action” involving a backup.
Where things start to get messy is when we need “action-parameters” - are they optional? This depends on the action. If we want to list filesets, we don’t need (or want) any “action-parameters”. If we want to list files withing filesets, we do. But if the “action-parameters” are not specified as options - i.e. parameters using the “–” prefix - they can become ambiguous, and the only way to remove the ambiguity is to make prior non-option parameters positional.
It is probably worth noting that solving these types of issues is why Unix/Linux command lines can get so messy/complicated!
So while I am not suggesting this as an actionable change to the existing duplicati-cli program, if I were re-implementing the program I would probably do something along these lines:
Separate commands for listing filesets and files. Or maybe make it “list={filesets|files|backups}”. Something to resolve the overload.
Specify “backup” object using options; either --target-url or even individual options for provider, username, password.
Allow backups to be identified by name - assumes/requires access to the appropriate server database which would have to be specified via option (If not using the default).
Maybe add a 2nd option similar to --parameters-file that would only be used to specify parameters to identify a backup and the passphrase? Not sure if it worthwhile…
In general define the syntax so that the only positional parameters that come after “action” are filenames. “help” would be an exception.
# DUPLICATI__REMOTEURL
# This is the remote url for the target backend. This value can be changed by
# echoing --remoteurl = "new value".
I’m thinking these (which can change various options) are safer, as there’s no CLI parser.
Posting in case anyone disagrees. Another discussion on CLI syntax and parsing is here.
Agree. It varies quite a lot. At the link I cited, I noted inconsistency even within Duplicati.
The issue in that discussion is more-or-less the same; positional vs optional arguments. Really the only way to resolve the issue is to avoid positional arguments as much as possible.
Another item I forgot to touch on is whether or not nonsensical arguments should be ignored or throw an error. In another post we ran into the issue where I was deleting filesets, and Duplicati was throwing an error because it wanted to delete expired filesets as well. The reason we ran into this was the way I was issuing the command - using commandline from the gui, removing the filename parameters, showing the options as text, and then deleting the --excludes while leaving the rest. Hence the “delete” command was being given the --retention option. One could argue that since you have to specify the fileset version(s) with the “delete” command, that it should not accept --retention, or at the least it should ignore it. I understand it is probably easier to parse by just accepting every option and ignoring what isn’t applicable.
public static async Task<int> Main(params string[] args)
{
RootCommand rootCommand = new RootCommand(
description: "Converts an image file from one format to another."
, treatUnmatchedTokensAsErrors: true);
MethodInfo method = typeof(Program).GetMethod(nameof(Convert));
rootCommand.ConfigureFromMethod(method);
rootCommand.Children["--input"].AddAlias("-i");
rootCommand.Children["--output"].AddAlias("-o");
return await rootCommand.InvokeAsync(args);
}
One approach would be to create a data structure containing all of the options and a list of commands for which they are valid. Use something similar to the above fragment to build a parser for the command list and any options that are global and set “treatUnmatchedTokensAsErrors” to false. In the method that gets called build a secondary parser for the options in the data structure that are specific to that command.
I’m sure someone with more knowledge and experience with modern Microsoft libraries could improve on this or come up with a better alternative.
But I think any of this still comes down to a breaking change, or a new program (duplicati-cli2 anyone?).
The parser was hand-written many years ago where there was no standard way for C#, so it is based loosely on my experience with Linux-based tools and argument parsing at the time.
Today, there are some excellent libraries for parsing commandline arguments, such as System.CommandLine from Microsoft. I plan to rewrite the code to use that library, so it is less accepting on common errors and follows more closely to what other programs will parse.
No, that one is replacing the URL after the commandline is parsed, so it is no longer positional. The way the CLI parsing code is made, it expects a list of positional arguments, and then inside each operation it will unpack if it needs a remote url.
We cannot really fix this without potentially breaking things. If there are anyone currently relying on the current behavior, like:
duplicati-cli backup dummy --arg1
Then the current logic is to replace dummy. If we instead insert it, it would fix what @jberry02 found, but then there is suddenly an unwanted argument.
Yes, it will be a breaking change for sure. But nice that you also found the System.CommandLine library.
Here “replace” means “replace the value in targetUrl with the new value”, so there is no index issue to address.
There is a process where it goes from the list to the structure.
I was considering that we could make the parameters-file and run-script both work on the parsed structure to avoid the indexing issue, but it would still not fix the issue experienced by @jberry02 as it requires a dummy value to provided, which is the thing that was not expected.
Then it is really down to minor differences in parsing, especially how the commandline GUI is translating into the resulting array that is sent.
It is possible that there is a place where it discovers an empty backend url and substitutes with the first source folder to be more flexible?