Rclone TEMP Problems

thommyX · October 27, 2024, 9:48pm

Dropbox, Google, Local, Webdav Backups work, but a new rclone has TEMP-Problems?

Details
Duplicati Version: 2.0.8.1_beta_2024-05-07
Windows 10, all available updates installed

RClone configuration:
Storage Type: Rclone
Local repository: local: (or local or x) => but is required!!!
Remote repository: Google
Remote path: duplicati

RClone is in the local PATH (for system, so for all users)
Path to config is set in ENV for all users

RClone knowns both repositories local: and Google:

C:\Users\server>rclone about local:
Total:   63.445 GiB
Used:    42.991 GiB
Free:    20.454 GiB

C:\Users\server>rclone about Google:
Total:   75 GiB
Used:    60.480 GiB
Free:    14.434 GiB
Trashed: 60.480 GiB
Other:   87.242 MiB

The backup source is available. dir \myserver\photo works.

Commandline

"C:\Program Files\Duplicati 2\Duplicati.CommandLine.exe" backup "rclone://Google/duplicati?rclone-local-repository=local:&allowed-ssl-versions=Tls12" "\\myserver\photo_r\\" --backup-name="google Photo " --dbpath="C:\Users\server\AppData\Local\Duplicati\BZSKXNQWOI.sqlite" --encryption-module="aes" --compression-module="zip" --dblock-size="200MB" --passphrase="mypass" --disable-module="console-password-input" --tempdir=C:\temp\

In the following output the backup has a problem with the normal temp folder (%TEMP% in the user appdata\local). Various other Duplicati-Backups work and save the temp files in this folder. Started a different backup (with Google Drive target and not rclone):

So I tried to change it for this backup to c:\temp\

The files are created but I think there is a problem sending this files to the Rclone remote?

//?/c:/Users/server/:c:/temp/dup…
c:/User/server is the “current dir” of the Duplicati process?

Changed directory to c:\temp and error message changes to this:

Backend event: Put - Started: duplicati-b7f4d10d760644553b10015dec7d01f98.dblock.zip.aes (1.32 MB)
Uploading file duplicati-b7f4d10d760644553b10015dec7d01f98.dblock.zip.aes (1.32 MB) …
Operation Put with file duplicati-b7f4d10d760644553b10015dec7d01f98.dblock.zip.aes attempt 1 of 6 failed with message: 2024/10/27 22:43:38 ERROR : Local file system at //?/C:/temp/:C:/temp/dup-574c5ffa-0c70-4112-bfbb-a29ca1d33d00: error reading source root directory: directory not found
2024/10/27 22:43:38 ERROR : Attempt 1/3 failed with 1 errors and: directory not found
2024/10/27 22:43:39 ERROR : Local file system at //?/C:/temp/:C:/temp/dup-574c5ffa-0c70-4112-bfbb-a29ca1d33d00: error reading source root directory: directory not found
2024/10/27 22:43:39 ERROR : Attempt 2/3 failed with 1 errors and: directory not found
2024/10/27 22:43:39 ERROR : Local file system at //?/C:/temp/:C:/temp/…

If I change local repository to empty the error changes to this:

“:C:\temp\dup-1e351d35-9532-42ae-99c4-95b6c072c0eb”:

thommyX · October 29, 2024, 7:04am

I solved it myself!

The problem ist the required field rclone-local-repository=…

The backup does not work either with

rclone-local-repository=local
rclone-local-repository=local:
rclone-local-repository=c:\temp\
rclone-local-repository=c:\temp
rclone-local-repository=

BUT: If I remove this parameter from the commandline then the backup works!

Problem now: How to get rid of this field in the Duplicati GUI? This field is MANDATORY and if I leave it empty I get an error when saving the configuration. So no possibility to remove it.

ts678 · October 29, 2024, 5:59pm

How are you doing those experiments, since you seem to use the GUI?

or do you do something like Export As Command-line for run and tests?

GUI run seems to work here with local as the GUI Local repository.

local: fails like yours. rclone debug log suggests how it gets that result:

2024/10/29 08:35:54 DEBUG : rclone: Version "v1.57.0" starting with parameters ["C:\\bin\\rclone.exe" "copyto" "local::C:\\tmp\\tempdir\\dup-96c59a72-9095-4873-b138-017cf86e413a" "onedrive:DuplicatiBackups/rclone/duplicati-20241029T123545Z.dlist.zip" "--log-file" "rclone.log" "--log-level" "DEBUG"]
2024/10/29 08:35:54 DEBUG : Creating backend with remote "local::C:\\tmp\\tempdir\\dup-96c59a72-9095-4873-b138-017cf86e413a"
2024/10/29 08:35:54 DEBUG : Using config file from "C:\\Users\\Maintenance\\.config\\rclone\\rclone.conf"
2024/10/29 08:35:54 DEBUG : fs cache: renaming cache item "local::C:\\tmp\\tempdir\\dup-96c59a72-9095-4873-b138-017cf86e413a" to be canonical "//?/C:/Duplicati/duplicati-2.0.8.1_beta_2024-05-07/RUN/:C:/tmp/tempdir/dup-96c59a72-9095-4873-b138-017cf86e413a"
2024/10/29 08:35:54 DEBUG : Creating backend with remote "onedrive:DuplicatiBackups/rclone/duplicati-20241029T123545Z.dlist.zip"
2024/10/29 08:35:56 ERROR : Local file system at //?/C:/Duplicati/duplicati-2.0.8.1_beta_2024-05-07/RUN/:C:/tmp/tempdir/dup-96c59a72-9095-4873-b138-017cf86e413a: error reading source root directory: directory not found
2024/10/29 08:35:56 DEBUG : One drive root 'DuplicatiBackups/rclone/duplicati-20241029T123545Z.dlist.zip': Waiting for checks to finish
2024/10/29 08:35:56 DEBUG : One drive root 'DuplicatiBackups/rclone/duplicati-20241029T123545Z.dlist.zip': Waiting for transfers to finish
2024/10/29 08:35:56 ERROR : Attempt 1/3 failed with 1 errors and: directory not found
2024/10/29 08:35:56 ERROR : Local file system at //?/C:/Duplicati/duplicati-2.0.8.1_beta_2024-05-07/RUN/:C:/tmp/tempdir/dup-96c59a72-9095-4873-b138-017cf86e413a: error reading source root directory: directory not found
2024/10/29 08:35:56 DEBUG : One drive root 'DuplicatiBackups/rclone/duplicati-20241029T123545Z.dlist.zip': Waiting for checks to finish
2024/10/29 08:35:56 DEBUG : One drive root 'DuplicatiBackups/rclone/duplicati-20241029T123545Z.dlist.zip': Waiting for transfers to finish
2024/10/29 08:35:56 ERROR : Attempt 2/3 failed with 1 errors and: directory not found
2024/10/29 08:35:56 ERROR : Local file system at //?/C:/Duplicati/duplicati-2.0.8.1_beta_2024-05-07/RUN/:C:/tmp/tempdir/dup-96c59a72-9095-4873-b138-017cf86e413a: error reading source root directory: directory not found
2024/10/29 08:35:56 DEBUG : One drive root 'DuplicatiBackups/rclone/duplicati-20241029T123545Z.dlist.zip': Waiting for checks to finish
2024/10/29 08:35:56 DEBUG : One drive root 'DuplicatiBackups/rclone/duplicati-20241029T123545Z.dlist.zip': Waiting for transfers to finish
2024/10/29 08:35:56 ERROR : Attempt 3/3 failed with 1 errors and: directory not found
2024/10/29 08:35:56 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         2.0s

2024/10/29 08:35:56 DEBUG : 4 go routines active
2024/10/29 08:35:56 Failed to copyto: directory not found

The rclone experts are at forum.rclone.org, but it looks to me like the extra colon confused rclone, causing it to invent the new path (similar to original post) that it couldn’t actually find (so an error).

Above output is from GUI run (mine works without the extra colon) with Options Advanced option rclone-option set to

--log-file rclone.log --log-level DEBUG (and you might prefer a full path for your log-file)

If I leave off the trailing colon (just enter local in GUI). it works for me. Are you running this way?

thommyX · October 29, 2024, 9:43pm

I use the GUI for the configuration and when I am done I use “Export backup configuration” => Command Line

This command line is copied into a software the I have written (a scheduler).

At the moment I do a manual edit for the one rclone task and remove the “?rclone-local-repository=removeme” (I entered removeme as local repo name).

Backup works since then and backuped 50 GB and I also tried a restore of some files that also worked.

This way is works, but it is anoying to remember every time to replace this if I chance something on this backup.

There is also no explaination in the documentation for the local repository? rclone can work without any problem with local paths! rclone x: remote:… for example will work if x: is a local drive in Windows. Also the other way rclone remote: x: or rclone x: y: if both are local drives.

Is this maybe a Linux thing with Duplicati and rclone on Linux?

ts678 · October 29, 2024, 10:51pm

I linked to it on my last post (though for a different option). Here’s what it says:

--rclone-local-repository (String) Local repository for Rclone. Make sure it is configured as a local drive, as it needs access to the files generated by Duplicati. Default value: local

however if you mean you don’t quite like the explanation, you can look at dialog below about it:

The forum is not a bug tracker, but if you like I guess you could open a UX or Features topic to perhaps improve this (maybe even get rid of it). That will see if it’s popular, but a GitHub Issue would have better tracking. There’s a lot more that could be done than there are people to help.

Does it run as the same user as your Duplicati (TrayIcon or Server)? Rclone config is per user.
My GUI export runs (unless I change users) on 2.0.8.1 Beta with local and no manual editing.

Maybe you could try the rclone debug log file option to see what rclone is saying for your error.

thommyX · October 30, 2024, 9:13am

With “no expaination” I meant: “No understandable explanation”. And yes: I found this second block were someone had a similar question. But also with no explanation. rclone can handle local paths as the paths are (without a local: repository). So c:\temp\ or x: or …

So I still do not understand the purpose of this local_repository-Parameter?

I will try to figure out this github stuff to create a suggetion to remove this parameter.

ts678 · October 30, 2024, 11:04am

Because there are way too many requests and other needs, odds are that a fix will take awhile.

You can try removing the problem with the current code by using the debug method I’ve shown:

doesn’t say how it does not work even at Duplicati level, but seeing rclone use would be better.

You can also try this to see if output looks like mine. I’ll concede my rclone is several years old:

C:\bin>rclone config show local
--------------------
[local]
type = local
--------------------

Another option would be to go to Google directly, as you said in OP that it works (as others do).

kenkendk · November 1, 2024, 7:44am

I am not well versed in how rclone works, but the code that requires the parameter in the UI is here:

github.com

duplicati/duplicati/blob/master/Duplicati/Server/webroot/ngax/scripts/services/EditUriBuiltins.js#L1191


      
          			res = EditUriBackendConfig.show_error_dialog(gettextCatalog.getString('Bucket name can only be between 3 and 63 characters long and contain only lower-case characters, numbers, periods and dashes'));
          		}
          		
          		if (res)
                      continuation();
              };
          	
              EditUriBackendConfig.validaters['rclone'] = function (scope, continuation) {
                  var res =
                      EditUriBackendConfig.require_field(scope, 'Server', gettextCatalog.getString('Remote Repository')) &&
                      EditUriBackendConfig.require_field(scope, 'rclone_local_repository', gettextCatalog.getString('Local Repository')) &&
                      EditUriBackendConfig.require_field(scope, 'Path', gettextCatalog.getString('Remote Path'));
          
                  if (res)
                      continuation();
              };
          
          	EditUriBackendConfig.validaters['cos'] = function (scope, continuation) {
          		var res =
                      EditUriBackendConfig.require_field(scope, 'cos_app_id', gettextCatalog.getString('cos_app_id')) &&
                      EditUriBackendConfig.require_field(scope, 'cos_secret_id', gettextCatalog.getString('cos_secret_id')) &&

The code that actually implements the rclone backend is here:

github.com

duplicati/duplicati/blob/master/Duplicati/Library/Backend/Rclone/Rclone.cs#L68


      
          }

          

          public Rclone(string url, Dictionary<string, string> options)

          {

              var uri = new Utility.Uri(url);

          

              remote_repo = uri.Host;

              remote_path = uri.Path;

          

          

              local_repo = "local";

              opt_rclone = "";

              rclone_executable = "rclone";

              /*should check here if program is installed */

          

              if (options.ContainsKey(OPTION_LOCAL_REPO))

                  local_repo = options[OPTION_LOCAL_REPO];

              if (options.ContainsKey(OPTION_REMOTE_REPO))

                  remote_repo = options[OPTION_REMOTE_REPO];

              if (options.ContainsKey(OPTION_REMOTE_PATH))

                  remote_path = options[OPTION_REMOTE_PATH];

I can see that there is some logic problem here, as setting an option to the empty string is not the same as excluding it. For your purpose, I think it would be sufficient to simply set the local repo to local as that is what the backend will do if it is missing.

If you set it to an empty string, it will send the remote path prefixed with : to the rclone executable:

github.com

duplicati/duplicati/blob/master/Duplicati/Library/Backend/Rclone/Rclone.cs#L205


      
          }

          

          

          public IEnumerable<IFileEntry> List()

          {

              String str_result;

          

              try

              {

          

                  str_result = RcloneCommandExecuter(rclone_executable, $"lsjson {remote_repo}:{remote_path}", CancellationToken.None).Await();

                  // this will give an error if the executable does not exist.

              }

          

              catch (FolderMissingException ex)

              {

                  throw new FolderMissingException(ex);

              }

          

              using (JsonReader jsonReader = new JsonTextReader(new StringReader(str_result)))

              {

kenkendk · November 1, 2024, 8:04am

I made a PR that fixes this: