Uri Class and other mechanisms try to interpret the URI, so maybe the left part is getting dropped here.
As evidence of that, Test connection for folder C:\later @ Sync shows an access attempted to the C:\ProgramData\Duplicati\duplicati-2.0.6.104_canary_2022-06-15\RUN\ Sync folder, which
could mean that the first part got dropped and the rest became a relative path into my current directory.
If somehow it still makes sense to keep this convention, you might be able to hide it from Duplicati with mklink /d so things look more normal to Duplicati while you still keep your favorite view. I haven’t tested.
EDIT:
Other supporting evidence for this was seeing the web browser POST a file: URL to server, specifically:
Not that I really agree with such naming practices as they always seem to end up conflicting with something a some point.
Side note/flashback: Any one remember using ALT+0255 to make a “hidden folder” in DOS?
I digress, it looks like you should be able to use the ~ (tilde symbol to the left of the 1/! key) in the same manner as the @ symbol. It will keep folders with it at the top of the list and Duplicati doesn’t seem to be affected.
I tried a few simple backup/restores and it seemed to be fine. I’d suggest testing it on your end to verify it does truly work for you but it looks promising to me.
I think the point of this is not if you agree with my naming practices or not, nor if you wonder why I would want to do this. I have good reasons for it and it has served me well for a long time and I’m not gonna change it, neither for the tilde sign or anything else. As long as a character is allowed by the OS to be used in the filename, any decent software dealing with said file system should be able to handle that.
I really thought this was a bug since every other backup or sync tool I have ever used, browser based or otherwise, paid or free have had no problem whatsoever to read or write from or to these folders. Syncthing, Goodsync, UrBackup, Dropbox, Google Drive, OneDrive, Acronis, well, every other similar software have no problem with this.
ts678:
I can see that the use of the @ can get interpreted as an URI in some cases, but I think software that is used for backup solutions on the file system should be able to make the distinctions for when to treat this as a folder path and when to see it as an URI.
Someone else may find the ~ suggestion helpful as an alternative that provides the exact same functionality as the @ symbol without causing any conflicts.
I don’t and I don’t, it’s about anyone else that reads this thread, knowing that it is a bad idea to use characters like @ in file or folder names regardless of what OS or software they use, even if they’ve done it forever and ‘everything else works fine’…
A quick internet search for “characters to avoid in filenames” shows that many other people aside from myself who think that even though you can use certain characters (like an @ symbol) maybe, just maybe, you shouldn’t because every once in a while (often at the worst time possible) you will find out it that it can mess things up and all you had to do to avoid it was use a slightly different character. Oddly, I did not find the ~ symbol on any of those lists of characters to avoid, strange stuff…
If everything else is obviously so much more capable than Duplicati, clearly you should be using those tools and not wasting your time around here with this seemingly incapable software and it’s crappy workarounds.
When configuring a backup, the storage URL parameter determines which storage provider will be used. Each storage provider has a different URL format which can be used along with the documented options below.
but Local folder or drive does have a confusing number of different ways shown, and some unshown.
The additional complication is that both the C# code in server and JavaScript in GUI try to deal with it.
is probably what the CLI is using, and \\?\C:\later @ Sync\ or file://C:/later @ Sync/ work, however trying to put them into GUI gets fields broken out by the JavaScript, and not as I would wish:
While there’s seemingly a bug in here somewhere, finding and fixing it without breakage may be hard because regular expressions are hard, and because Duplicati has other URI uses that must still work.
The forum is not a bug tracker, but you are welcome to open an Issue. Also see if any is already filed, because typically ones that affect a lot of people and have no workarounds are good ones to work on.
Please also keep in mind that Duplicati is created by people who volunteer personal time to try to help. Volunteer developers are needed. Forum is doing better, but your help and understanding are needed.
Having rejected some proposed workarounds, what about this one mentioned before, and now tested?
C:\>mklink /?
Creates a symbolic link.
MKLINK [[/D] | [/H] | [/J]] Link Target
/D Creates a directory symbolic link. Default is a file
symbolic link.
/H Creates a hard link instead of a symbolic link.
/J Creates a Directory Junction.
Link Specifies the new symbolic link name.
Target Specifies the path (relative or absolute) that the new link
refers to.
C:\>mklink /d "C:\Duplicati Backup" "C:\@ Backup"
symbolic link created for C:\Duplicati Backup <<===>> C:\@ Backup
C:\>
Note that the icon shows that it’s a link, however Test connection follows that to @ Backup at top.
Adding the synonym lets you get view you like, while also giving Duplicati something it handles well.
Ideas are probably all that we can give you instantly. If you wish, file an Issue, to get in the long line.
Done.
EDIT:
So here’s my test backup in a folder whose naming you favor:
JimoboJones is correct to an extent. Eg Android 12 on my phone it itself won’t allow use of ? in the folder names for file creation but works fine for folder creations (when I tested it in app code). The system file explorer app won’t even allow ? in the folder name.
On desktops, various applications have seen issues with symbols in paths as problems people have seen. This stuff happens. You only happened to have not seen it until now lol.
It is however something that programmers working on paths should think about checking but its understandable to not think about it or test every variation given how much time one wants to spend on testing and can think of.
It is likely Duplicati with a bug though. Although, the .net version or whatever could see it too and be the reason. But, its expected to be the application.
Duplicati can backup and restore a source path that starts with, contains or ends with an @ symbol without issue.
The issue only presents when using an @ symbol in the destination path.
Symptoms:
Any part of the specified path prior to the @ symbol is removed. The job will save (which it shouldn’t) and run but then goes to “Verifying backend files.” almost instantly then a minute or so later errors out complaining about an invalid path along with error message in the GUI showing the specified path contents after the @ symbol.
Work Arounds:
Use a different character in the destination path.
Create a symbolic link and use the linked path in Duplicati.
Fix:
A quick look at that Regex says it’s not directly responsible for removing the @ symbol, likely one of the .replace() methods a bit further down is stripping it out. Probably just need another if to check if it’s a local destination or URI and if so run a slightly different sanitizing on the string. Probably quite low on the priority list but it’s also probably a fairly easy fix.
A third workaround (but probably more unappealing to GUI users) is to use CLI. These one-line
regular expressions are hard to read. Thanks to @JimboJones for persevering with code reads.
I got as far as finding by experiment a CLI URI syntax that worked, then GUI JavaScript choked.
In addition to another one-line regex there, one would have to read JavaScript more than I do…
I wasn’t willing and able to either put a debugger onto the program or write test code for parsing,
given how there are a couple of workarounds. I did take the C# regex and try to study multi-lines.
organized somewhat around what gets grabbed via (?<name> and what’s optional (? after something). Syntax section of Uniform Resource Identifier Wikipedia article has a diagram, and links to references.
An optional userinfo subcomponent followed by an at symbol (@), that may consist of a user name and an optional password preceded by a colon (:).
was my theory of where the portion before the at symbol was disappearing into. It IS sometimes used. Storage Providers shows some examples of what looks like the userinfo part. I’m not sure if a folder accepts it or not. It wouldn’t apply for truly local, but could SMB credentials maybe be given that way?
Someone could, if willing, continue my use of the Backup destination three-dot menu to put URIs in.
I kept getting unwanted pieces landing in the Username and Password fields, but I don’t do JavaScript.
Upon a deeper dive, it looks like the regex is excluding (not capturing) the @ symbol when dealing with file:// URI paths.
It works correctly for capturing usernames in http:// URIs such as user.name@example.com but excludes the :// and @ parts when capturing file:// URIs which in turn creates too many slashes in the path and the capture groups end up capturing the wrong matches and populating the other fields with slashes or a slash and the drive letter.
If file://./ is used to specify the host it blows up when an @ is added to the path, group 2 becomes ./driveletter which doesn’t work, using file:/// causes a null in group 4 that which seems fine until an @ is added and it falls apart, group 2 becomes /driveletter.
I didn’t notice anything about passing SMB credentials over file:// I expect that to be the OSes job to be deal with but I’ll have to do some more reading to be sure.
As always Windows requires some special cases to work around like supporting dual or quad slashes (currently does not seem to support either) but that’s a side note and it just means a few more optional ? capture groups and an even longer expression.
Looks like the expression needs a bit of love, I’ll see what I can come up with.
Good point, but if they have to go to the OS, Duplicati needs to be able to pass them, not kill them.
The on-screen Username and Password fields are (we think) for SMB use. You can see remarks at Local disk credentials issue where someone thought it was to change the user, for NTFS ACL use.
Thank you for trying to help, unlike others around here.
As I’ve said before, the naming convention has a long history for me and is used in many software solutions and is used in many of my Python and AutoHotkey script. It is therefor an enormous task to undergo to change all this, which I have neither the time nor desire to do for a change in backup tool.
Maybe I will try this software at a later stage as it seems to be a good option, and I do appreciate all the voluntary done by the people behind it. It just doesn’t seem to be for me right now.
Sorry to barge in your research, but I’m always interested in understanding why this is this way. While I did not look at the code, I am reminded of the @ in sftp uri and thinking that maybe it was the motivation behind this special handling. It may be that the code is using a generic regex to handle all sorts of backends uri, but that writing such a generic regex may be too complicated, and that the way forward could be to handle the whole problem differently - a specific handling for some backends possibly.
That’s mere speculation so feel free to ignore my ramblings.
I haven’t tried this to see what would actually happen with Duplicati, but a warning to anyone who might attempt it with any sort of UNIX/Linux: “~xxxxxx” generally translates automatically to “path to user xxxxxx’s home directory” (Like “/home/xxxxxxx”). While it might work in Duplicati (if neither Duplicati, nor any system libraries it uses, parses it that way), anyone trying to access such a folder/file from a UNIX/Linux shell is going to, at least, have to escape the “~” to reference it. I know this thread isn’t about alternatives, and this is not a suggestion to the OP since it won’t help them. Underscore (“_”) is about the only special character I’d ever use in a file/folder name. It’s harmless in both UNIX/Linux and Windows.
(py311) gerard@j5005:~/Images$ mkdir ~mydir
(py311) gerard@j5005:~/Images$ echo "this is my file" >~mydir/~myfile.txt
(py311) gerard@j5005:~/Images$ ls ~mydir
'~myfile.txt'
(py311) gerard@j5005:~/Images$ ls ~/~*
ls: impossible d'accéder à '/home/gerard/~*': Aucun fichier ou dossier de ce type
(py311) gerard@j5005:~/Images$ ls ~/Images/~*
'~myfile.txt'
Protecting things from shells is always a challenge. GUI needs less help, but has its issues, e.g. here…
Thanks to @JimboJones for finding that old one. I tried 2.0.5.1 and 2.0.4.3, but they also didn’t decode.
One reason I tried is that sometimes things do change, e.g. here’s a change done in Sept 2020 in area:
A more recent fun one was a fix of “AllowEncodedSlashes On” for Apache (the reverse proxy?) to work.
I don’t know if the formal URI standards give enough guidance to change URI parsing, but we’d need to avoid breaking on URIs that are already used in backups. The concern was mentioned in the above PR.
@gpatel-fr I’m not sure what you’re trying to say, your example certainly works. If you try that sequence when there is a user named “mydir” on the system it won’t turn out the same way.
I am just warning people that “~” can have unexpected side effects in UNIX/Linux.
point is, the restrictions on ~username have nothing to do with Unix/Linux. This is a behaviour of special programs, shell. Most (not all) conform to the old Posix standard of aliasing ~username to the home directory of ‘username’. But this does not apply to all software, including Duplicati, and all the servers that can run under Linux or some other Unixy systems, that will accept such directory names without any problem.
Also file managers such as Dolphin (the Kde file manager) do not follow this Posix convention.
Even Bash will accept to handle them if one asks nicely:
(py311) gerard@j5005:~/Images$ mkdir '~root'
(py311) gerard@j5005:~/Images$ cd ~root
bash: cd: /root: Permission non accordée
(py311) gerard@j5005:~/Images$ cd '~root'
(py311) gerard@j5005:~/Images/~root$ pwd
/home/gerard/Images/~root
