When I choose google drive as my backup location, I’m able to get a working test connection when I specify no path. However, if I try to specify any path (whether it exists on my google drive account already or not), I get Failed to connect: Internal Server Error.
How can I diagnose what’s going on? For now I’m just uploading directly to the root directory but this isn’t ideal.
Welcome to the forum @nrgbistro
This appears to be an internal server error from Google. Hard to know reason.
What OS is this, and does a simple path (e.g. just a name, no slashes) error?
From Windows (and probably from anything), a backslash gives me this error:
and if I try a similar test using Duplicati.CommandLine.BackendTool.exe list with a URL from an
Export As Command-line (then edit that to use a backslash, making it illegal), I get the same error.
If I try to get more information using not-so-easy Windows .NET Framework network tracing, I see:
{
"error": {
"errors": [
{
"domain": "global",
"reason": "invalid",
"message": "Invalid query",
"locationType": "parameter",
"location": "q"
}
],
"code": 400,
"message": "Invalid query"
}
}
after finding the detailed response in the unencrypted data, and putting it all back together to post here.
In this case, no additional information, but there’s a chance (maybe not great) your case will see more.
Google Drive: 500 internal server error
were some 2018 cases, and the road to success was to wait for Google to fix the internal server error.
There were some other ideas there that may or may not help you resolve whatever’s failing at Google.
is possibly significant (but maybe only to Google). Duplicati by default shouldn’t even be able to see any folder that it didn’t create. This is a Google security feature. You can wind up with the same folder twice.
You should be able to see in the details box at the right of your screen that Duplicati created the files, e.g.
and you can compare that with the creation of the folder you’re trying to get to, to see if it would be seen,
but quite possibly the processing never got as far as looking at the characteristics of the specified folder.
Is this a simple personal Google Drive account? That’s what I use. I don’t know much about fancy types.
@ts678 I’ve got the exact same problem on 2.0.6.3_beta_2021-06-17 and Ubuntu 20.04. This makes Google Drive a bit unusable. I see nothing in my logs but in the HTTP response I get
{
"Error": "missing-folder"
}
Of course the folder does exist. I tried with slashes, without, without pre-creating the folder… But nope. Connection and backup is fine when field is empty though.
IMO this is a bug since Duplicacy has no problem backuping to a specific pre-existing folder on Drive.
What created it? Duplicati’s default is a more secure option where it only sees folders it created.
Please check your folder. One created by Duplicati looks like
What you should do is not manually make folder, but use Test connection to let Duplicati do it.
If you prefer to use an option that Google wants to kill (as insecure) get an AuthID using this site:
https://duplicati-oauth-handler.appspot.com/
Google’s planned cutoff (which they have not done yet, and I have no idea whether they ever will):
Enhancing security controls for Google Drive third-party apps
sounds like Duplicati didn’t make it, but you can verify. Below is the Duplicacy (not Duplicati) issue:
more restrictive scope for Google Drive #555 (asks for drive.file scope, like Duplicati is already on)
EDIT:
I’m still not certain what scope Duplicacy uses, but I’m pretty sure it’s not (or was not?) drive.file.
Reducing Google Drive scope: a simple proposal
Google Drive, drive.appdata scope, service account impersonate
I see so this works for Duplicacy because of the full access scope ? Or a more permissive one anyway. Well in the meantime I can confirm Duplicati does not create the folder so I don’t know what to to do. Giving it full access does not seem like a good solution to me given what you said.
Testing to personal (not Business, Education, etc.) plan. What’s yours?
Testing Path on server to see what I can create with nothing there now.
foldercreate Test connection says “The folder foldercreate does not exist. Create it now?” Yes does.
foldercreate Backup creates the folder and puts the backup in it.
foldercreate/subfolder Test connection says “The folder foldercreate/subfolder does not exist. Create it now?” Yes does.
foldercreate/subfolder Backup creates the folder and puts the backup in it.
So this is one of the storage types that can automatically create for me. I think some require help.
Implication from the above is that if it couldn’t see your pre-created folder, it should create its own.
Google Drive has the unusual ability to have multiple folders or files that all share the same name.
But apparently something went wrong, so a close look at what was done (and resulted) is needed.
What does it do, e.g. from Test connection? Does it message like I said? How do you respond?
If you are in a complex plan involving sharing, permissions may be complex. I only have personal.
Presumably you have storage space of your own. You could just try a test exactly like I performed.
Duplicati’s job editor will insist that you give it at least one file. I gave it a short text file for this test.
More technical is to test Drive with Duplicati.CommandLine.BackendTool.exe to see what it will do.
You can get a storage URL from Export As Command-line.
Here’s a look in the top-level folder I made. If I delete URL foldercreate, I see Duplicati’s folders.
What I don’t see (because this is a limited view) are non-Duplicati items of personal “home” folder.
Duplicati.CommandLine.BackendTool.exe list "googledrive://foldercreate?authid=REDACTED"
Name Dir/File LastChange Size
subfolder Dir 1/1/0001 12:00:00 AM
Doing list should be very safe, but there are other operations possible that actually change data.
Testing to personal (not Business, Education, etc.) plan. What’s yours?
I have a personal Drive of 15Go, nothing fancy, all default stuff. There’s already some data in it but that shouldn’t matter.
What does it do, e.g. from
Test connection? Does it message like I said? How do you respond?
That’s in the first post. I get "Failed to connect: Internal Server Error " in the popup, and the actual HTTP response is code 500 with the body from my first post. It only works if I leave the server path empty, but it creates the backup at the root of the drive, which isn’t what I want.
You could just try a test exactly like I performed.
I’m using the linuxserver/duplicati Docker image and I can’t find a similar command line tool in the container.
EDIT: I just found this Failed to connect: | Google Drive · Issue #38 · linuxserver/docker-duplicati · GitHub but it does not help a lot
This combination sounds strange, but I don’t get to this level much (no nice logging, unfortunately).
We’re also unfortunately back to the mystery of the 500…
Unless they removed it, it’s with the detailed Duplicati installation (no wrapper script like in /usr/bin).
Possibly /usr/lib/duplicati is their installation directory. If you find it and it won’t start, try saying mono
before .exe file path. Some Linux systems need that hint.
Meaning even a simple one-word-no-slash Path on server fails I assume. If so, that seems weird, especially since file uploads must be working if you can put them where you don’t want them in root.
If you can find the tool, createfolder after a URL edit seems to be the method to create the folder.
I’m pretty sure Test connection here is a list, so that’s the first test to try to see if it gets the 500.
I just tried a list, changing createfolder to foldermissing, and at least the tool quietly said that
Command failed: The requested folder does not exist
It might tempt you to try a test with a full access token, but that could possibly go after the other testing.
I’d note that lots of storage services don’t even offer a restricted scope to keep apps in their own files…
Oooh actually it’s only the “Test Connection” that does not work ! It does create the folder (even subfolder) fine when the job is started. Since Test Connection does not create the folders I guess my error make sense. Maybe Duplicati could add a warning or something ?
What exactly happens when you press “Test connection”? If I try that with my Internet off, it pops Error:
Failed to connect: Failed to authorize using the OAuth service: The remote name could not be resolved: ‘duplicati-oauth-handler.appspot.com’. If the problem persists, try generating a new authid token from: Duplicati OAuth Handler
I think there’s already some level of error reporting now. I know of no way to force Google to error 500.
I’m still pretty sure this Test connection (unlike a few fancy ones) is a list. Ever find BackendTool?
While I’d like to understand the Test connection issue, bottom line might be – does this fix backup?
When I put a path, exactly what I said before: “Failed to connect: Internal Server Error”.
I’m still pretty sure this
Test connection(unlike a few fancy ones) is alist. Ever find BackendTool?
Found it !
abc@fda341668241:/app/duplicati$ mono Duplicati.CommandLine.BackendTool.exe list "googledrive://foldercreate?authid=REDACTED"
Name Dir/File LastChange Size
Command failed: The requested folder does not exist
Note that even if I create /foldercreate in my drive. I still get the same error. However this works :
abc@fda341668241:/app/duplicati$ mono Duplicati.CommandLine.BackendTool.exe createfolder "googledrive://foldercreate?authid=REDACTED"
It’s just that Test Connection shouldn’t take into account the server path field IMO
Actually backup probably always worked, I was just trying with Test Connection the whole time while setting up the actual backup job. Thinking it didn’t work I never bothered trying just running it. This is just an UX issue.
If you do that through the Google web UI, the result is expected. You can also check file info:
for any folder that you click to see Google Drive’s data on who (you or Duplicati) created that.
This proposal would defeat the whole idea of testing whether destination folder already exists.
There are two parts to it, it appears. One is whether a connection at all to server can be done.
The other part is whether that folder exists already. If not, you are asked if you wish to make it.
I think there’s at least one storage type (I forget which) which relies on such manual creations.
The two aspects seem (to me) to fit together, and testing exactly the desired path seems best.
I’m almost certain that the folder test infers that the connection works, without specific testing.
Not solved yet. The error 500 seems wrong, but you now have some folders Duplicati created.
What happens if you Test connection on the foldercreate you made using BackendTool?
You can also see if you can list it. There “should” be no subfolder in it, unless you made one.
There “should” also be no errors coming back. If this works, then you can test with a subfolder.
The goal with all this is to see if things work more as expected when you’re not right at the root.
Unfortunately I can’t easily test a root with no Duplicati-created things in it, if that state matters.
If you have a Windows system, comparison tests between that and Docker may be interesting.
Windows can also do unencrypted network tracing so you can catch some actual server traffic.
This gets rather technical. I’d give it a try, except I can’t reproduce the issue that you’re having.
It might provide a logical explanation for why Google sent error 500. That’s still quite a mystery.
EDIT:
Maybe you have a more advanced network troubleshooting ability that works on your Docker?
Where did the response show up, and if you can get the response, can you get the request as well?
Be careful about posting things. We wouldn’t want any authentication or credential information seen.
IMO from a user point of view, Test Connection, as its name implies, should test the connection, not try to create the folders. But yeah the ideal scenario is what you propose, since it involves asking the user about folder creation.
Test Connection works with a path (say testfolder) if I use BackendTool createfolder prior yes. Listing works too. If I add a subfolder like testfolder/sub, I get the 500.
In my browser network tab. And sure:
POST /api/v1/remoteoperation/test HTTP/2
Host: duplicati.<REDACTED>
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:100.0) Gecko/20100101 Firefox/100.0
Accept: application/json, text/plain, */*
Accept-Language: fr,fr-FR;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate, br
Content-Type: application/json;charset=utf-8
X-XSRF-Token: <REDACTED>
Content-Length: 88
Origin: https://duplicati.<REDACTED>
DNT: 1
Connection: keep-alive
Referer: https://duplicati.<REDACTED>/ngax/index.html
Cookie: default-theme=ngax; AUTHP_SESSION_ID=<REDACTED>; access_token=<REDACTED>; xsrf-token=<REDACTED>
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
TE: trailers
googledrive://notcreatedfolder?authid=<REDACTED>
HTTP/2 500 Internal Server Error
cache-control: no-cache, no-store, must-revalidate, max-age=0
content-language: fr
content-type: application/json
date: Tue, 31 May 2022 19:23:51 GMT
server: Caddy
server: Tiny WebServer
set-cookie: xsrf-token=<REDACTED>; expires=Tue, 31 May 2022 19:33:50 GMT;path=/;
content-length: 34
X-Firefox-Spdy: h2
{
"Error": "missing-folder"
}
Actually I know what is happening here… It ends up in this line duplicati/Duplicati/Server/WebServer/RESTMethods/RemoteOperation.cs at beaf03562fdcf4425e962085bdf7175d6a465f49 · duplicati/duplicati · GitHub ! My guess is it tries to list files in a non existent folder (because Test Connection does not create them prior). For some reason the call fails and Duplicati is saying yeah folder is missing. Error response code is a bit dramatic though for sure 
It does both. Because you seem technical (care to volunteer some? Duplicati benefits from volunteers):
https://github.com/duplicati/duplicati/blob/master/Duplicati/Library/Interface/IBackend.cs
shows what a backend is supposed to be able to do, which is actually a bit more than the remote itself:
The backends encapsulate the actual communication with a remote host with a simple abstraction, namely that the backend can perform 4 operations: GET, PUT, LIST, DELETE. All operations performed by Duplicati relies on only these operations, enabling almost any storage to be implemented as a destination.
As mentioned earlier, testing is usually the list operation. At least one backend includes a fancy test.
Duplicati.CommandLine.BackendTester.exe is an even fancier (slower) one that one can run by hand.
Cloud destinations typically use some sort of an API, and there’s not really a separate connection test.
Connection test with list from the API seems a good choice, testing connectivity, authentication, API.
What simpler could there be? At a TCP level, connecting means little, and sometimes IP isn’t known…
If you’re arguing that the API list should target the root, maybe, but what UX would help with folder?
If you can lay out what UI flow you have in mind, that would help, but you might then have to code it.
There are far too few develper-volunteers, and things that are merely non-ideal tend to not get done.
I wonder if the original post did this too? We’d been trying to puzzle out why Google Drive sent 500.
Here, we don’t know what it sent, but we know what Duplicati sent. How’s your JavaScript? Look at:
and although I don’t read JavaScript (my C# reading is a bit better), this is presumably what should have posed the question of what you want to do about the missing folder. To show it visually, I’m getting below:
but you don’t? Do you get an error popup? When talking about UX, please say how the UI is working.
Looking under the covers is fine for debugging though. I put Wireshark on this too to look a the traffic.
That won’t work on the network going to Google, because it’s encrypted.
Sorry C# isn’t my thing
I don’t think Google is doing it, Duplicati is. It’s in my previous link. See the method implementation where it sets HTTP 500 duplicati/Duplicati/Server/WebServer/RESTMethods/RequestInfo.cs at beaf03562fdcf4425e962085bdf7175d6a465f49 · duplicati/duplicati · GitHub
The existing one is fine (asking the user if he wants to create it). You just have a bug in the JS you linked. I put a breakpoint in the JS debugger :
See ? The message variable does not have the right value.
Code should be var message = data.data.Error instead. If I override the message variable in the JS console to “missing-folder” and resume the program, I do get the popup asking me to create the folders, and if I say yes it works as expected and the test is successful. By the way, that bug should totally fuck up the other ifs as well 
You should totally be able to replicate the issue. Maybe you’re not testing with the same version ? As previously said, I’m on 2.0.6.3_beta_2021-06-17. Super recent.
You cited some, so I was hoping… The good news is that you know JavaScript. More about that below.
I agree, and I wrote “Here, we don’t know what it sent, but we know what Duplicati sent.”. Contrast to “I wonder if the original post did this too? We’d been trying to puzzle out why Google Drive sent 500.” last August from the original troubleshooting, and I’m not sure where OP looked. Few look at HTTP traffic…
Completely unable in 2.0.6.3_beta_2021-06-17. I was using Edge before. Now trying Firefox. It works.
I’d have expected a bug like “Test connection” being as bad as it seems for you to be widely reported. Maybe different storage types work differently, that cuts it down, and maybe OP was seeing issue too?
Issues typically go to development by opening a GitHub Issue, but if you know GitHub, consider a Pull. JavaScript developers in either area may be even rarer than C# volunteers. Any other experts around?
Is something (maybe a proxy) rewriting your messages from Duplicati? I have a direct connect, and get
HTTP/1.1 500 missing-folder
Cache-Control: no-cache, no-store, must-revalidate, max-age=0
Content-Language: en-US
Date: Tue, 31 May 2022 20:13:44 GMT
Content-Length: 36
Content-Type: application/json
Server: Tiny WebServer
Connection: close
Set-Cookie: xsrf-token=REDACTED; expires=Tue, 31 May 2022 20:23:43 GMT;path=/;
...{
"Error": "missing-folder"
}
per Wireshark. Tiny WebServer is likely Duplicati web server. How did server: Caddy get in yours?
I don’t know JavaScript, but Response.statusText seems perfect for my response but not for yours.
Possibly relevant to your response is the HTTP version. I’m getting 1.1 and you’re getting 2.
Setting custom statustext in express response using default message?
says the “reason phrase” is gone in HTTP 2, so response body should be used. RFC says:
8.1.2.4. Response Pseudo-Header Fields
HTTP/2 does not define a way to carry the version or reason phrase
that is included in an HTTP/1.1 status line.
That’s exactly what I was writing Yes my Duplicati is sitting behind Caddy, which is a web server/reverse proxy which runs on HTTP2. My take on this is that Duplicati should rely on some kind of standard error codes in the response body, not on reason phrases. Which should not have been relied on in the first place, as pointed out by a Caddy maintainer here (exact same scenario).
The path of least resistance would be to make the change on the message variable I suggested earlier I think. The change should be transparent… ? I can open an issue if you want.
That line was rolled out in 2015, possibly thinking of the looser language of the 1999 standard:
6.1.1 Status Code and Reason Phrase does sound like the intent was just a human-read note.
Maybe whoever wrote the code didn’t keep up with the firmer rules from the 2014 RFC update.
Unless someone makes a point of doing this, I can easily see a lot of older information around.
So I see Caddy Dec. 21 was saying they had no option (blaming the Go lib) but rewrite reason.
The person asking said it was a deal-breaker with Caddy and went looking for an alternative…
I wonder how many other places need a change? At least it’s finite. A text search says around 16.
That’s counting ones like the one here, and there are a couple more statusText usages around.
If we’re lucky, there will be a standard HTTP style where the message we need is in the body too.
I’m just searching *.js with Notepad++, but the GitHub search box also looks quite good at finding.
https://github.com/Pectojin/duplicati-client is a command line client in Python running the web API.
It’s by a former Duplicati developer who reverse-engineered it. Maybe you can check how it works.
You could, but it might not get done due to lack of help. A pull request might have a better chance.
There’s also some automated testing (which may or may not do GUI/API work) during submission.
Another way to test these things might be to start modifying/testing JavaScript installed on-system.
I found some JavaScript files in my installation folder under webroot\ngax\scripts\directives
Well, we made some progress de-mystifying a couple of things. Thanks for help. More may help…