Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]

avmaksimov · September 5, 2022, 7:56pm

Hello. I want to backup my home folder on Ubuntu. But I mounted some drives via rclone. So I can’t enter this via “Source Data” page when edit the backup (“Failed to process the path: Access to the path ‘/home/andrei/yandex-disk’ is denied.”) and also I saw the similar warnings in logs:

2022-09-05 21:00:04 +03 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: /home/andrei/yandex-disk/

2022-09-05 21:00:04 +03 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-PathProcessingError]: Failed to process path: /home/andrei/yandex-disk/

2022-09-05 21:00:04 +03 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: /home/andrei/mail.ru/

2022-09-05 21:00:04 +03 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-PathProcessingError]: Failed to process path: /home/andrei/mail.ru/

2022-09-05 21:00:05 +03 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: /home/andrei/yandex-disk/

2022-09-05 21:00:05 +03 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-PathProcessingError]: Failed to process path: /home/andrei/yandex-disk/

2022-09-05 21:00:05 +03 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: /home/andrei/mail.ru/

2022-09-05 21:00:05 +03 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-PathProcessingError]: Failed to process path: /home/andrei/mail.ru/

I added /home/andrei/mail.ru/ and /home/andrei/yandex-disk/ (also without trailing slash) to excludes but have no success. I added --allow-missing-source=true but the problem still exists.

What can I do to remove these warnings?

gpatel-fr · September 6, 2022, 8:35am

I have no personal experience with this particular feature, but if you want to backup data that you don’t have access to as an user, a workaround is to run Duplicati as a service and it will run as root, so that should take care of that.
If you prefer ignore the directories you don’t have access to, an exclude should work. If you export as command line, is your include instruction like

--exclude="*/home/andrei/mail.ru*/"

ts678 · September 6, 2022, 4:19pm

Can you manually access the mounted files when running as the same user Duplicati is running as?
Rclone forum would be a better place to look into rclone mount (which provides permission options).

avmaksimov · September 6, 2022, 4:54pm

yes, I have access to this folder without any problem. But D2 can’t: so I can’t open this folders, but I can’t open it via e.g. Files.
I think the problem is D2 running from the user who can’t access to this folder. But it’s not needed).

avmaksimov · September 6, 2022, 5:43pm

It doesn’t help. D2 starts as root, but I mount as user. So there is no access for root.

I’m a linux newbie, so sorry about may be stupid questions. So I think there are some solutions:

turn off the warning in D2 (the best way for me!)));
mount as root (but I think may be I will stump with another access problem);
give root user some access to this mounted folder (I don’t know how);
run D2 as user “andrei” (I think it’s not a good idea).

I don’t like neither 2-4 ways, but is exist another way?)

avmaksimov · September 6, 2022, 7:40pm

My answers are:

I don’t know how. Please help me))
It’s strange but neither D2 nor I can’t access to this folders.
I gave the access for group but after mounting group change to “andrei” (. Also I tried “chown andrei …” but also without success.
I don’t like it. But is if the only way?

gpatel-fr · September 6, 2022, 8:14pm

I see that rclone mounts with FUSE, so indeed it’s session specific.
As I said, if you don’t need them the easy fix is to ignore them. I have already given you the syntax in my first reply.

ts678 · September 6, 2022, 8:47pm

In GUI, you enter it on Source Data page Filters three-dot Edit as text without quotes and with - in front. Alternatively you could try the Exclude folder choice and see what it builds. Some builders are odd…
Using filters and The TEST-FILTERS command are available too. Filters aren’t the easiest things to set.

avmaksimov · September 7, 2022, 6:56pm

Thank you. As I wrote before i have already used excludes but tested your one and without success (.
I don’t want to ignore it)).

avmaksimov · September 7, 2022, 6:57pm

It’s strange but there are no any warnings in console versions. May be it just a bug?)

Btw, if I can’t open folder in GUI because some inner folders are inaccessible it have to be a bug too).

ts678 · September 7, 2022, 7:43pm

Console versions of what, and what did you do? If you went to GUI to do Export As Command-line,
maybe you should just sanitize private information and post the rest so we can read actual exclude.
What user was this no-warnings test done as? Remember that access is allowed to mounting user.

The suggested exclude was maybe more than necessary, but it seems to work, per test-filters:

$ find /tmp/andrei
/tmp/andrei
/tmp/andrei/homefile
/tmp/andrei/mail.ru
/tmp/andrei/mail.ru/mailfile
$ duplicati-cli test-filters /tmp/andrei --exclude="*/tmp/andrei/mail.ru*/"
Including source path: /tmp/andrei/
Excluding path due to filter: /tmp/andrei/mail.ru/ => (*/tmp/andrei/mail.ru*/)
Including path as no filters matched: /tmp/andrei/homefile
Matched 1 files (0 bytes)
$ duplicati-cli test-filters /tmp/andrei --exclude="/tmp/andrei/mail.ru/"
Including source path: /tmp/andrei/
Excluding path due to filter: /tmp/andrei/mail.ru/ => (@/tmp/andrei/mail.ru/)
Including path as no filters matched: /tmp/andrei/homefile
Matched 1 files (0 bytes)
$

however there’s a chance the problem is that (it looks like) Duplicati glances at the excluded folder:

$ strace -f -qq -e signal='!all' -P /tmp/andrei/mail.ru mono /usr/lib/duplicati/Duplicati.CommandLine.exe test-filters /tmp/andrei --exclude="/tmp/andrei/mail.ru/"
Including source path: /tmp/andrei/
[pid  4284] lstat("/tmp/andrei/mail.ru", {st_mode=S_IFDIR|000, st_size=4096, ...}) = 0
[pid  4284] lstat("/tmp/andrei/mail.ru", {st_mode=S_IFDIR|000, st_size=4096, ...}) = 0
Excluding path due to filter: /tmp/andrei/mail.ru/ => (@/tmp/andrei/mail.ru/)
Including path as no filters matched: /tmp/andrei/homefile
Matched 1 files (0 bytes)
$

For local files, lstat works if the containing folder is accessible, even if the specified folder is not, as

$ ls -ldn /tmp/andrei/mail.ru
d--------- 2 1000 1000 4096 Sep  7 15:00 /tmp/andrei/mail.ru
$ ls -ln /tmp/andrei/mail.ru
ls: cannot open directory '/tmp/andrei/mail.ru': Permission denied
$

I don’t know exactly how rclone mount access handles this, but it’s possible that it works differently.

avmaksimov · September 8, 2022, 6:23pm

Hello and thank you.
I mean that I exported as command line and then replace to test-filters. I see no problem.

So today I’ve created a new backup with new structure only for test. Also I dismount “Yandex Disk” from my standard location to test one.

After mounting “find” started to display all files inside yandex-disk folder.

There are test-filters result:

$ mono /usr/lib/duplicati/Duplicati.CommandLine.exe test-filters /home/andrei/test/source --exclude="*/home/andrei/test/source/yandex-disk*/"
Including source path: /home/andrei/test/source/
Excluding path due to filter: /home/andrei/test/source/yandex-disk/ => (*/home/andrei/test/source/yandex-disk*/)
Including path as no filters matched: /home/andrei/test/source/folder/
Including path as no filters matched: /home/andrei/test/source/file.txt
Including path as no filters matched: /home/andrei/test/source/folder/file-in-folder.txt
Matched 2 files (0 bytes)

$ mono /usr/lib/duplicati/Duplicati.CommandLine.exe test-filters /home/andrei/test/source --exclude="/home/andrei/test/source/yandex-disk/"
Including source path: /home/andrei/test/source/
Excluding path due to filter: /home/andrei/test/source/yandex-disk/ => (@/home/andrei/test/source/yandex-disk/)
Including path as no filters matched: /home/andrei/test/source/folder/
Including path as no filters matched: /home/andrei/test/source/file.txt
Including path as no filters matched: /home/andrei/test/source/folder/file-in-folder.txt
Matched 2 files (0 bytes)

But:

$ sudo mono /usr/lib/duplicati/Duplicati.CommandLine.exe backup file:///home/andrei/test/dest /home/andrei/test/source --backup-name=test --dbpath=/usr/lib/duplicati/data/ENHFPSNGMA.sqlite --encryption-module= --compression-module=zip --dblock-size=50mb --no-encryption=true --disable-module=console-password-input --exclude="*/home/andrei/test/source/yandex-disk*/"
[sudo] password for andrei: 
Backup started at 9/8/2022 9:14:46 PM
Checking remote backup ...
  Listing remote folder ...
Scanning local files ...
  0 files need to be examined (0 bytes)
Error reported while accessing file: /home/andrei/test/source/yandex-disk/ => Access to the path '/home/andrei/test/source/yandex-disk' is denied.
Failed to process path: /home/andrei/test/source/yandex-disk/ => Permission denied
Error reported while accessing file: /home/andrei/test/source/yandex-disk/ => Access to the path '/home/andrei/test/source/yandex-disk' is denied.
Failed to process path: /home/andrei/test/source/yandex-disk/ => Permission denied
Checking remote backup ...
  Listing remote folder ...
Verifying remote backup ...
Remote backup verification completed
  Downloading file (817 bytes) ...
  Downloading file (691 bytes) ...
  Downloading file (1.41 KB) ...
  Duration of backup: 00:00:00
  Remote files: 3
  Remote size: 2.88 KB
  Total remote quota: 90.22 GB
  Available remote quota: 29.27 GB
  Files added: 0
  Files deleted: 0
  Files changed: 0
  Data uploaded: 0 bytes
  Data downloaded: 2.88 KB
Backup completed successfully!

$ sudo mono /usr/lib/duplicati/Duplicati.CommandLine.exe backup file:///home/andrei/test/dest /home/andrei/test/source --backup-name=test --dbpath=/usr/lib/duplicati/data/ENHFPSNGMA.sqlite --encryption-module= --compression-module=zip --dblock-size=50mb --no-encryption=true --disable-module=console-password-input --exclude="/home/andrei/test/source/yandex-disk/"
Backup started at 9/8/2022 9:15:46 PM
Checking remote backup ...
  Listing remote folder ...
Scanning local files ...
  0 files need to be examined (0 bytes)
Error reported while accessing file: /home/andrei/test/source/yandex-disk/ => Access to the path '/home/andrei/test/source/yandex-disk' is denied.
Failed to process path: /home/andrei/test/source/yandex-disk/ => Permission denied
Error reported while accessing file: /home/andrei/test/source/yandex-disk/ => Access to the path '/home/andrei/test/source/yandex-disk' is denied.
Failed to process path: /home/andrei/test/source/yandex-disk/ => Permission denied
Checking remote backup ...
  Listing remote folder ...
Verifying remote backup ...
Remote backup verification completed
  Downloading file (817 bytes) ...
  Downloading file (691 bytes) ...
  Downloading file (1.41 KB) ...
  Duration of backup: 00:00:00
  Remote files: 3
  Remote size: 2.88 KB
  Total remote quota: 90.22 GB
  Available remote quota: 29.27 GB
  Files added: 0
  Files deleted: 0
  Files changed: 0
  Data uploaded: 0 bytes
  Data downloaded: 2.88 KB
Backup completed successfully!

I added “sudo” due the error: “Failed to load connection with path ‘/usr/lib/duplicati/data/ENHFPSNGMA.sqlite’. => Unable to open the database file”

So I decided to run test-filters with sudo and have the same error:

$ sudo mono /usr/lib/duplicati/Duplicati.CommandLine.exe test-filters /home/andrei/test/source --exclude="*/home/andrei/test/source/yandex-disk*/"
Including source path: /home/andrei/test/source/
Error reported while accessing file: /home/andrei/test/source/yandex-disk/ => Access to the path '/home/andrei/test/source/yandex-disk' is denied.
Error reported while accessing file: /home/andrei/test/source/yandex-disk/ => Access to the path '/home/andrei/test/source/yandex-disk' is denied.
Failed to process path: /home/andrei/test/source/yandex-disk/ => Permission denied
Failed to process path: /home/andrei/test/source/yandex-disk/ => Permission denied
Including path as no filters matched: /home/andrei/test/source/folder/
Including path as no filters matched: /home/andrei/test/source/file.txt
Including path as no filters matched: /home/andrei/test/source/folder/file-in-folder.txt
Matched 2 files (0 bytes)

$ sudo mono /usr/lib/duplicati/Duplicati.CommandLine.exe test-filters /home/andrei/test/source --exclude="/home/andrei/test/source/yandex-disk/"
Including source path: /home/andrei/test/source/
Error reported while accessing file: /home/andrei/test/source/yandex-disk/ => Access to the path '/home/andrei/test/source/yandex-disk' is denied.
Error reported while accessing file: /home/andrei/test/source/yandex-disk/ => Access to the path '/home/andrei/test/source/yandex-disk' is denied.
Failed to process path: /home/andrei/test/source/yandex-disk/ => Permission denied
Failed to process path: /home/andrei/test/source/yandex-disk/ => Permission denied
Including path as no filters matched: /home/andrei/test/source/folder/
Including path as no filters matched: /home/andrei/test/source/file.txt
Including path as no filters matched: /home/andrei/test/source/folder/file-in-folder.txt
Matched 2 files (0 bytes)

And:

ls -ldn ~/test/source/yandex-disk
drwxrwxr-x 1 1002 1002 0 сен  8 20:47 /home/andrei/test/source/yandex-disk
ls -ln ~/test/source/yandex-disk
total 15596
-rw-rw-r-- 1 1002 1002 4349583 янв  7  2018  20171231_164959.jpg
<other folder and files>

But:

$ sudo ls -ldn ~/test/source/yandex-disk
[sudo] password for andrei: 
ls: cannot access '/home/andrei/test/source/yandex-disk': Permission denied
$ sudo ls -ln ~/test/source/yandex-disk
ls: cannot access '/home/andrei/test/source/yandex-disk': Permission denied

So the reason of the warning is clear. But why D2 tries to enter the excluded folder… I don’t understand it.

avmaksimov · September 8, 2022, 7:31pm

eeeee… I’ve just found the solution…\

Unfortunately I can’t avoid D2 bug (i’m sure it’s a bug) but I’ve found a solution for rclone:

Uncomment “user_allow_other” option in /etc/fuse.conf .
Run rclone mounting with “–allow-other” option (“–allow-root” option doesn’t work anymore).

Now I have no problem with it. I hope this solution will help someone!))

ts678 · September 9, 2022, 1:01am

It doesn’t, but it does look at what the file is (using lstat). Linux says it’s allowed, but FUSE rejects.
After digging through code again, I found that I started down the same tracing path two years ago:

Ignored folders are not really ignored

There are lots of checks on the path before it’s allowed in. Include/exclude is done after lstat check.
Let me do a check based on my old work, and you can try the same thing for Yandex FUSE mount.

# id
uid=0(root) gid=0(root) groups=0(root)
# ls -ln /run/user/1000
ls: cannot access '/run/user/1000/doc': Permission denied
ls: cannot access '/run/user/1000/gvfs': Permission denied
total 0
srw-rw-rw- 1 1000 1000   0 Jun 19 15:45 bus
drwx------ 3 1000 1000  60 Jun 19 15:45 dbus-1
drwx------ 2 1000 1000  60 Sep  7 19:57 dconf
d????????? ? ?    ?      ?            ? doc
drwx------ 2 1000 1000 140 Jun 19 15:45 gnupg
d????????? ? ?    ?      ?            ? gvfs
drwx------ 2 1000 1000 100 Jun 19 15:45 keyring
drwx------ 2 1000 1000  80 Jun 19 15:45 pulse
drwxr-xr-x 2 1000 1000  80 Jun 19 15:45 systemd
# stat /run/user/1000
  File: /run/user/1000
  Size: 240       	Blocks: 0          IO Block: 4096   directory
Device: 33h/51d	Inode: 2           Links: 10
Access: (0700/drwx------)  Uid: ( 1000/     ted)   Gid: ( 1000/     ted)
Access: 2022-09-08 20:51:00.604002609 -0400
Modify: 2022-07-05 19:48:20.505349706 -0400
Change: 2022-07-05 19:48:20.505349706 -0400
 Birth: -
# stat /run/user/1000/gvfs
stat: cannot stat '/run/user/1000/gvfs': Permission denied
#

So that’s the explanation, and I’m glad you found an insecure workaround (too bad about allow-root).

avmaksimov · September 9, 2022, 7:05am

Of course, I’ll try your solution if you give me.

I see that the problem in D2 is in the fact, that for the fist it scans the sources and only after it excludes or includes. That’s I call the bug)). Not a mistake, a logical problem. Do you know a profit for this solution? If D2 excludes folders before the scan would finish quickly and a backup too.

ts678 · September 9, 2022, 7:11pm

I didn’t say I had one. I did ask you to test to see if your case fails like mine, but my guess is yes.
To be specific, you could (as root) test if stat /home/andrei/test/source/yandex-disk fails.
You probably established that with ls -ldn though, although I’m less confident how that queries.

It starts with a FUSE bug, IMO, breaking normal Linux behavior, although one can argue about it.
It definitely gives an access error for an unlisted case, but that might just be how things go now…

Changing Duplicati (if it can be done safely, and that’s a developer call, not mine) would be good.
Maybe testing for a block special file (such as disk) is done first to try to reduce damage risk later.

Because the forum is not a bug tracker, could you see if the other topic filed this issue? If not, file.

Documented stat behavior (see lstat man page, which seems to hold at least locally) is generous.

These functions return information about a file. No permissions are required on the file itself, but-in the case of stat() and lstat() - execute (search) permission is required on all of the directories in path that lead to the file.

$ id
uid=65534(nobody) gid=65534(nogroup) groups=65534(nogroup)
$ ls -ldn mail.ru
d--------- 2 1000 1000 4096 Sep  7 15:00 mail.ru
$ ls -ln mail.ru
ls: cannot open directory 'mail.ru': Permission denied
$

But under FUSE, permissions apparently ARE required on the file itself, adding a new way to error.
Regardless, it would be good for Duplicati to allow the user to manually exclude the problem folder.