Since my attempt configuring a Duplicati task to backup from my new NAS (DS220+) to my old one (DS211j) failed (see this thread), I thought I’d reduce complexity by just plugging in a USB-stick as a backup target.
In docker I mounted one of my shared folders (/Programs) as a source folder with path /NAS20/Programs.
I mounted the target path as expected: /usbshare1 → /USB-target
Now when I configure a task i Duplicati I can specify the target path as a subfolder on my USB stick, like /USB-target/dup and successfully test connection.
For the source path however I can only specify /NAS20/Programs, but no subfolders. A little yellow attention sign in the “Programs” name indicates what a run confirms with warnings: “2022-09-21 17:52:41 +02 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: /NAS20/Programs/”
When looking at the filesystem in a terminal in docker (as root), I see full permissions (777) for the USB-target and below (probably because it’s FAT32), but no permissions (000) for the Programs folder and below (I use BTRFS on my NAS)
In DSM my user “duplicati” has read/write permissions for the docker folder and read-permissions for all other shared folders.
Considering the missing permissions it is clear to me that traversing the source folder in a backup task can’t work - but what on earth am I missing here?
Any help is appreciated!
PS: So far I also failed to get Syncthing running in docker on DSM, so in general I may to be missing a fundamental thing in setting up the docker paths …
While I am using the native Duplicati for DSM, I do have a number of other things running in Docker containers. I’m no Docker expert, but I may be able to help (if in no other way than experimenting on my own setup). How did you set up your containers? Through the DSM Docker GUI or by hand? If by GUI, maybe include a screen shot? If by hand, maybe share the contents of the config file for the container?
Are you using the linuxserver image or the official duplicati image? The linuxserver one has a stricter stance on security so is a bit more challenging to get working correctly. Personally I choose to “trust” duplicati and use the official image, which runs as root and is easier to get working.
Thanks @Lerrissirrel and @drwtsn32 for your suggestions.
I installed the docker image strictly folowing the instructions on mariushosting. so the package seems to come from ghcr.io/linuxserver/duplicati.
My Linux background has mostly faded away over the past decades, so please bear with me when I ask silly questions like: Where do I find the config file for the container?
I thought that all this docker stuff is a simple click and run job suitable for Windows GUI guys, that’s why feel a bit at a brick wall right now.
I also read in some previous post that running Duplicati in docker is preferable over the native version, esp. since I am on DSM 7. Please correct me if I’m wrong. I am happy to switch if this makes things easier…
Thanks, this information helps a lot. I tried to do an install following these instructions, and it worked for me, but 1) I can see a couple of ways it might go wrong and 2) I’m still running DSM 6, so there may be an important difference for you at 7 that I don’t know about.
Does the user (whose PUID you used) have permissions to access /NAS20/Programs? You mention that it does, but let’s just double check. Does /NAS20/Programs (or just /NAS20) show in Control Panel->User (select user, click “Edit”) under “Permissions”? Note that that I’m not sure if setting read/write permissions on the upper level directory (NAS20/Programs) allows the user to inherit those permissions on all sub directories or not. You may need permissions on everything underneath, too, if files under /NAS20/Programs have been created by other users.
Or you can add your user to the “administrators” user group and it will inherit those group level access rights regardless of what your users permissions are.
And just to check the mount points are set up properly … what did you specify on these three lines?
-v /volume1/docker/duplicati/config:/config
-v /volume1/docker/duplicati/backups:/backups
-v /volume1/docker/duplicati/source:/source \
Alternately
You can go to the DSM Docker application, click on “Container” (on the left, between “Overview” and “Registry”)
find your Duplicati container in the list, right click on it and select “Details”
In the bottom left, click on “Volume” (which is between “Port Settings” and “Links”)
Either screen shot the window, or copy/list the rows under “File/Folder Mount Path Type”. From what you’ve said, you should have one row that says “/NAS20/Programs /source rw”
Lastly, maybe just running “df” from the terminal will give us information. Do you see a line showing something mounted at /source? If not, the problem is likely there.
I admit that in my installation, I don’t worry too much about security, so be aware that adding the user to “administrators” or using the “root” user, etc, will leave things less secure than they could be, but as drwtsn32 mentioned, it’s SO much easier to not have to worry about these things if you don’t need to!
I know this is a lot of text/questions, if any is not clear, just ask.
Thanks a lot for your guidance - I was already thinking in the direction of your first point.
At installation time - and strictly following instructions without really understanding implications - I specified PUID and GUID of the current user, which at that time was my admin account.
On the same occasion I created a new user duplicati on the NAS becasue I thought it’s a good idea to have control over hwat this guy can and cannot do. This user is member of its own group (g_duplicati) from which it gets its permissions (rw for the docker folder and ro for all other shared folders) and of course the default user group (100). It is not member of the the admin group (101).
Actually, I have no clue now how the ID I set up at installation and the duplicati account go together. I have the sneaky feeling they don’t …
After a closer look at Marius’ installation instructions I see that the installation is performed as “root” (General - > Create Task). Is this just a necessity for an installation or does it determine in which user-context the container is run later on?
Just for clarification When I open a Terminal (bash) in the running container, “whoami” tells me I am “root”. Is this the “big-root” of the NAS or a “root” only in the container environment? The latter seems to be the case, as I am not able to navigate up to /volume1 for example. So “root” in a putty session and “root” in a container shell are two different things - confusing (at first).
Since we’re talking about a home setup with NAS next to my desk I am willing to sacrifice security in favor of simplicity of the setup.
So for a test I put my user duplicati intoto the admin group, but this didn’t make any difference.
The more I think about it the more I get confused about the different users:
NAS-user duplicati: seems to have no effect whatsoever, since I specified a different user at installation time
assuming that now the container runs under my admin account - what would I need to do differently to account for this?
then there is this password I set up for accessing the docker website at :8200: I assume this is completely unrelated to the NAS-internal accounts.
the account I specify when setting up the target in the job configuration on the website: no matter what I enter here (admin, duplicati, nothing, nonsense,…) the connection test is successful (remember: it’s a USB-stick at the front-panel of the NAS). And likewise: in all cases I get no acceess to folders below /NAS20/Programs, when setting up the source.
Finally some more anwsers to your questions:
from the exported docker configuration (docker.json) here are the lines about the mount points:
I don’t quite follow your question about what is mounted at “/source”: What I have mounted to /source is what was specified during installation, which is /docker/duplicati/source.
/NAS20/Programs is a mount-point inside the container, while /Programs is the shared folder that I expect to have visible (= mounted) inside the container under this very mountpoint. (to be precise: /volume1/Programs).
I am wiling to delete the whole docker container and install again, with a different user, if this helps.
But still I’d rather better understand the effects of any changes…
Again, I’m no expert, but from what I understand 1) Yes, root (admin) must be the one to do the installation. 2) The user context of the application within the container is set by the PUID. 3) The container itself is running on the DSM as “root”
Since a docker container (at least in this case) is its own image if Linux, when you use the Terminal from within DSM’s Docker application, you are logging into that container’s Linux. So technically it’s only the root within the container. (although see further below as to how UIDs and user permissions affect things between DSM itself and the container.) If you have set your DSM up to allow ssh (I assume this was your Putty reference) then yes, this will be different - you’re logged into the Linux instance of DSM itself. So here you would see /volume1 for example.
What you noticed about not seeing /volume1 from the container Terminal is because /volume1 is a filesystem within your DSM. The only parts of your DSM’s filesystem(s) that you can see from within the container are the ones that you map as part of the container config. For instance, take this part of the config:
This is saying "Take /volume1/Programs from DSM and mount it at the location /NAS20/Programs within the container. Note that DSM’s Docker seems to play some games with the /volume1 part which you had to specify in the original container creation script, but you don’t see it here.
Correct, unless you specified the user ID of this user as PUID in the container setup, your “duplicati” user is not involved in any of this.
(skipping next question for now!)
Yup, completely unrelated, this is just the password to access the Duplicati GUI
That’s interesting. I suspect that Duplicati just does not use the target user name and password for “local folder or drive” targets. Going back to your original desired use of your other NAS, I imagine this would be required if you used ftp, for instance. I think we can ignore it here.
You can ignore this question. The other info you supplied answered what I was looking for. I think the instructions you followed intended the user to mount their desired source directory (The location within the DSM from which you want to backup) under “/source” in the container but you’ve mounted yours inside the container as /NAS20/Programs which is perfectly fine.
So we’re back to this question which, as far as I can tell, is the crux of the matter. First off, the container itself is running within DSM as “root” but it turns out that’s not very relevant here, either.
Some points on “users”:
There is a set of “users” inside the container’s Linux instance that have no relation to your DSM’s users (“cat /etc/passwd” in the container Terminal to see them)
DSM Docker will map user IDs inside the container to the same user IDs in your DSM (regardless of what the user names are either in the container or on your DSM)
As such, any files living on your DSM that are read/written from within the container are done so as the DSM user to which the container user’s UID maps. Conveniently, “root” maps to “root”.
When you connect to the terminal of the container, you are “root” (user ID 0) within the container, and so anything you read/write (or view with “ls”) in any DSM directories you’ve mounted within the container (like “NAS20/Programs”) will be allowed using the same rules as if you were “root” on your DSM.
Duplicati, on the other hand, is running in the container as your DSM “admin” user’s UID (mine’s 1024, I assume yours is too). You see this with “grep /etc/passwd” in the container terminal. You’ll get one line of output some other user name at the very left but with UID 1024. For me the user in the container was “abc”. Then if you “ps -ef | grep ” you will see the Duplicati processes running as “abc”. Or just “ps -ef | grep Duplicati” and look at the user name on the left. Then you can do "id and you’ll see the user ID matches the one you used for the PUID originally (your ‘admin’).
Duplicati, within the container, is trying to access “/NAS20/Programs” (which is really /volume1/Programs in your DSM) with the permissions that your “admin” user has. So let’s see if your issue really is permissions related.
From within DSM you can go to “Control Panel” → Users → right click on “admin” and in the window that pops up, click “Edit” → the “Permissions” tab. Find “Programs” in the list on the left, then look at the rest of the line to see what permissions the “admin” user has. For Duplicati, in the container, to see /Programs you’ll need to set at least “Read only” but if security isn’t an issue, just go with “Read/Write”.
If you find that your “admin” user did not already have “Read only” or “Read/Write” access, make the change, and then I suggest stopping/starting the container to be safe.
If “admin” already had “Read only” or “Read/Write” access to “/Programs” let us know. I’ll come back with some more things to check. One thing I’ll ask now since DSM plays some games there, do you still have what you used for the “Task” that you used to create the container? Specifically the lines with the “-v” would be helpful in validating everything.
on a sidenote: USB-target (the mountpoint of the usb stick) is owned by somone who as no entry in /etc/passwd of the container: drwxrwxrwx 8 1024 users 4096 Jan 1 1970 USB-target
In /etc/passwd of the DSM however I find that 1024 is the default system user (“admin”). Seems like this ownership is handed through to the container.
BTW following security advise, I disabled this default admin user and instead currently use an account with a different name as active admin (1026).
So it looks as if my DS-admin (1026) is handed over to the container as container-user abc (1026) ? Would make sense, because this corresponds to the PUID/GUID I specified at installation.
Interestingly enough, while /source is owned by abc (the installer owner), the point where I mounted my /volume1/Programs folder is owned by (container-) root.
This makes sense, because I mounted the folder after installation.
Considering that /source is owned by user abc, I mounted /volume1/Programs as /source/NAS20-Programs - hoping it would inherit the permisions from /source.
Unfortunately not:
root@7f3506bad028:/source# ls -l
total 0
d--------- 1 root root 1448 Sep 25 10:55 NAS20-Programs
which leads to the same effect as before: I can’t step into this folder in the duplicati job configuration, when I want to specify the source.
However all the relevant folders below this one (i.e. those that I see on Windows under Programs) are again owned by abc!
Here are the processes run by abc, or the “Duplicati” processes resp. Looks OK to me:
Lastly: Checking permissions in the user configuration on DSM confirmed, that my admin user has r/w permissions everyhwere - as expected for a user in the admin group.
And finally: Here is the complete script that I ran in the task, as suggested by Marius:
docker run -d --name=duplicati
-p 8200:8200
-e PUID=1026
-e PGID=100
-e TZ=Europe/Berlin
-v /volume1/docker/duplicati/config:/config
-v /volume1/docker/duplicati/backups:/backups
-v /volume1/docker/duplicati/source:/source
–restart always Package duplicati · GitHub<<<< I don’t know how to prevent the editor from turning this into a link
tl;dr - I know, so thanks if you followed my winding thoughts so far!
Now my conclusion of all this is:
mountpoints that I add from the Docker UI in DSM are created and therefore owned by (Docker-)root.
mounted folders can only be traversed (i.e. containing folders accessed in source setup of duplicati job) if they are owned by the user that was specified at container creation (here: abc)
These are contradictory conditions to me.
And the only ways out of this - that I see - don’t make sense to me: Either:I’d have to setup once and for all the mountpoints with -v option an container creation time, with no way to change them later. I didn’t try this yet, as I can’t believe this. Why would you then have the options to configure the mountpoints in the volume dialog after all? Or: manually fiddle with permissions after creating the mountoints (chown, chmod) or even set up (container-)root as the user at container creation (PUID=GUID=0) . Both seem like insane options to me.
Well, we’re both learning a bit about Docker and DSM
When you are connected to the container’s terminal, have you actually tried "cd"ing or "ls"ing inside, what is now, /source/Programs? In the test container I created, if I just “ls -l” in / my “source” folder shows up with “d---------” permissions too. but if I do “ls -l /source” I can still see everything inside there.
When you are mounting DSM directories inside the container after configuration, how are you doing that? If from the command line within the container, can you show the command?
One other thing I’ve been meaning to check just to make sure. When you go to add the sources in your backup definition, in the folder heirarchy, are you looking under the “Computer” part, to find your mount point (in my example, it’s “sources” and you can see some of the things under there from my “/volume1/Backups” directory (which I mounted as /source).
I ask, because when I created the container to experiment and went to go see if I could see my “source” directory, I momentarily thought "Oh, maybe, somehow with this Docker setup/instructions it knows to show what is in “/source” (in the container) under “Source Data” which I quickly realized was not the case.
(Edit: forgot to include this picture:)
Otherwise, assuming we’re still dealing with a permissions issue… You mention that you disabled your default “admin” user (good idea!) and use a different user with admin rights (UID 1026). Just to be clear, when you said “my admin user has r/w permissions everywhere”, were you referring to your admin user with UID 1026, or the DSM “admin” user?
Your Linux skills seem to be much better than you first hinted at, so apologies for going through things that you already know, and asking some “silly” questions. At this point, I cannot think of many possibilities for the problem except something “silly”.
Yes - and I’m glad having a companion on this trip
yes, here is the result (output of last ls command abridged, just to show different types of permissions):
root@7f5360baa028:/# cd source
root@7f5360baa028:/source# ls -l
total 0
d--------- 1 root root 1448 Sep 27 07:55 NAS20-Programs
root@7f5360baa028:/source# cd NAS20-Programs/
root@7f5360baa028:/source/NAS20-Programs# ls -l
total 67596
d--------- 1 abc users 694 Oct 19 2021 ‘1AUDIO-player, analyzer, codecs’
---------- 1 abc users 1910386 Dec 3 2002 ‘Omni Page Guide Ger.pdf’
drwxrwxrwx 1 root root 82 Sep 27 07:55 @eaDir
drwxrwxrwx 1 root root 790 Jul 5 17:20 ‘#recycle’
drwxr-xr-x 1 root root 1218 Sep 27 17:00 ‘#snapshot’
I can further cd and ls into the directories, and the picure is similar: all files and dirs owned by abc, permissions: none. (No more root-owned dirs further below, as expected).
I don’t use cli for mounting, I do it from the “Volume” tab in the “edit” dialog of the (stopped) container.
I read something about different types of mounts (bind, volume, tempfs), but since I have no way to specify the type from the GUI, I assume this doesn’t matter, or is preconfigured correctly.
In short: I have no clue what the actual command-line looks like that DSM executes when mounting.
I look under “Computer” and my “NAS20-Programs” folder is decorated with a tiny little yellow attention mark, indicating I can’t expand this any further (btw: no filters or excludes specified). The *.txt file that I just created with “touch” is displayed correctly, but this is logical, as it is owned by root:
As soon as I select something under /souce, the item is mirrored under this mysterious “Source data” folder at the bottom.
The interesting thing is: Where does this name “Source data” come from? I don’t have a folder by that name in my container-root (/config). And before I mounted my NAS20-Programs under /source, this folder didn’t show up there.
So this /source thing does seem to be something special as soon as it is not empty. A clue for digging further?
I was referring to my (one and only active) DSM-admin user, which is 1026. Due to being member of the admin group, it has r/w permissions everyhwere.
I’m glad they are gradually coming back to me, after quite some man-page reading and web-browsing
Quite a revelation was when I (re) discovered how to use puTTY from and peep into the file system from my Windows machine, next when I discovered the bash-terminal for exploring the container file-system. Befere this was just a nice black box with seven seals to me…
After working long years in SW tech-support; I acquired a certain stubborness when it comes seemingly “unsolvable” problems
We’ve checked a lot of things, but this still looks a lot like a file/directory permissions configuration issue on DSM. More on that below. First I want to clarify something I may have created confusion on.
“Source data” is a list of those areas that you have selected for backup. So it is exactly true that what you select under “User data” or “Computer” then gets displayed under “Source data”.
Getting back to permissions, this is still the only thing I can think of that can cause what you are seeing. So let’s try looking at them from a different angle. As I mentioned, I’m on DSM 6 so what you see may be a little different. Let’s try using File Station. In my example I’m checking my “Backups” directory on DSM. It sounds like the one you care about is “Programs”. Right click on “Programs” and select “Properties”.
First, let’s click on “Advanced Permissions”. In my case, no boxes are selected.
Next, click on the “Permissions” tab. You’ll see something like this, although some subset of your users, of course.
Click the “Advanced options” drop down (to the right of the Create/Delete/Edit buttons) and select “Permission Inspector”.
In the “User or group” box, select the user whose UID you used when setting up the Docker container (your admin user from what you’ve described). Here, I’m showing my “plex” user just for example.
Then look at the “Read” and “Write” sections. For things to work with the Duplicati Docker container you should see check marks like in this picture. Duplicati should actually work just with “Read” permissions, but let’s make sure both Read and Write are set for simplicity. In your case, since it’s your admin user, you may also see the “Administration” boxes checked too and that’s ok.
However, if you DO see all of the Read and Write boxes checked, then I am quickly running out of ideas of what your problem is!
One more note on permissions and what you see with “ls” from the container’s: As far as I can tell, DSM’s Docker implementation is doing some special translation between it’s notion of access control and the container’s “linux” access controls. You can see some evidence of this by looking at the output of “mount” within the container’s terminal. Again, my “Backups” DSM folder is mounted as “/source” in the container. The “synacl” mount option is almost certainly something custom that Synology did for their Docker implementation.
As a result, I don’t think looking at “ls -l” within the container really means anything. For example, in my own test, I have my “Backups” folder mounted as “/source” and ls -l of / shows “source” as “d----------” but Duplicati can still see everything underneath /source.
And lastly (for now), one other thing I’d like to double check - if/when you have changed permissions on “Programs” within DSM as part of your experimenting, how have you “refreshed” Duplicati’s view of the permissions? From my own experimentation, a “reload” of the Duplicati gui page is required for it to see the permission changes. I had previously tried clicking on one of the other circles in the “Add backup” sequence (like “Destination”) then back to “Source Data” and that is not enough for Duplicati to notice the changes.
For anyone else who may be following this thread if you have any more ideas of what might be going on here or ways to check how things are configured, please don’t hesitate to offer them up. I’m close to running out of ideas!
Ah - shoot. My bad: I changed UI language before I made the screenshot, and then got surprised by this “source data” because before it used to be “Quell-Daten” (whose meaning was clear to me before from other jobs I am successfully running). It was then the “source” in the name which tricked me into believing this would be mysteriosuly related to my /source directory…
Same here for “Programs”
In the permissions inspector of my “Programs” folder I see nothing unexpected:
My DSM-admin user (in fact the whole admin group) has all r and w permissions checked here (no admin check-boxes however)., FWIW since I had lately added my “duplicati” user also to the admin group, this guy has the very same permissions
agreed. As I mentioned before: despite all folders listed as “d----------” I can I further cd and ls into the directories. The crux is, that duplicati obviously can not
My desperation is/was not deep enough yet to fiddle with permissions of shared folders. But anyway, I always got back to “Home” first in the Duplicati GUI or even set up a new job configuration to check the effect of a change (mounts etc). Furthermore I haven’t observed inconsistent behavior after several reboots of my WIndows as well as my DSM systems over the past days …
On a different note: For a try I installed the duplicati container from the duplicati repository in parallel (under /volume1/docker/duplicati2), via the DSM-docker GUI. However it was a bit of poking in the dark, as I have no clue if this package requires a different setup. So I did more or less the same as with the linuxserver distribution: manually created and mounted /backups, /config and /source, moved XDB_CONFIG_HOME to /config, left the local port to be set automatically - and hoped for the best. The container somehow started up with no complaints, but that’s about it. In the terminal things look abit different, as there is no user “abc”, but instead the DSM-admin’s UID (1026) is listed directly in the permissions. I see no log file, nor can I connect from my windows system via http://nas:<local port>/. This only brings up the message “The host header sent by the client is not allowed”.
I don’t want to open another discussion on this here - just saying I’m ready to try another distribution if this would simplify things.
I still have a feeling we’re overlooking something simple but I have no more ideas as to what it could be.
One more note on DSM permissions, though: User permissions take precedence over group permissions. You’ve already said that your admin user has appropriate permissions and that the admin group does too. But one other place that gives you a useful view of the “effective” permissions is in Control Panel → Shared Files → right click on “Programs” → select “Permissions” tab and see something like this:
Again, you’ve said you’ve already checked this stuff, but this particular view is convenient in that it shows both the users permissions and the users group permissions plus, in the “Preview” column, it shows the effective permissions. I made this example with my “admin” user to show a case where the group does have access but the user explicitly does not have access and so “admin” does not have access.
Like I said, I am out of ideas at this point and I don’t like giving up either!
On the topic that you did not want to open here… You may already have realized it, but the second Docker container would not be able to get port 8200 (since the first Duplicati container has it) and so would have defaulted to something else. I’ve also run into some cases of Duplicati denying access to its web interface. While I don’t remember the scenario or reasons, you may want to try your NAS’ IP address instead of its name (or the name if you tried the ip adress) when accessing from your Windows system browser.
My bad: I changed UI language before I made the screenshot, and then got surprised by this “source data” because before it used to be “Quell-Daten” (whose meaning was clear to me before from other jobs I am successfully running). It was then the “source” in the name which tricked me into believing this would be mysteriosuly related to my /source directory…