Restoring File ownership and permission

lukics · February 4, 2023, 5:41pm

I am running Duplicati inside a docker container using this docker-compose.yaml:

version: '3'
services:

  duplicati:
    image: duplicati/duplicati
    container_name: duplicati
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Europe/London
      - CLI_ARGS= #optional
    volumes:
      - /datavol/duplicati:/config
      - /mnt/backup/duplicati:/backups
      - /datavol:/source/datavol
      - /home/lukics:/source/home
    ports:
      - 8200:8200
    restart: unless-stopped

user lukics has the ID 1000

when I backup this folder content:

-rw-rw-rw- 1 lukics lukics  173 Feb  3 13:45 printer_data/config/KlipperScreen.conf
-rw-rw-rw- 1 lukics lukics   51 Feb  2 06:33 printer_data/config/README.md
-rw-rw-rw- 1 lukics lukics 1938 Jan 26 16:17 printer_data/config/crowsnest.conf
lrwxrwxrwx 1 lukics lukics   41 Sep 22 02:24 printer_data/config/mainsail.cfg -> /home/lukics/mainsail-config/mainsail.cfg
-rw-rw-rw- 1 lukics lukics 2466 Jan 26 16:19 printer_data/config/moonraker.conf
-rw-r--r-- 1 lukics lukics 7733 Feb  4 14:58 printer_data/config/printer.cfg
-rw-rw-rw- 1 lukics lukics  770 Dec 31 15:34 printer_data/config/sonar.conf
-rw-r--r-- 1 lukics lukics    0 Feb  4 18:10 printer_data/config/test.cfg
-rw-r--r-- 1 lukics lukics    0 Feb  4 18:20 printer_data/config/test2.cfg
-rw-r--r-- 1 lukics lukics    0 Feb  4 18:28 printer_data/config/test3.cfg
lrwxrwxrwx 1 lukics lukics   60 Sep 22 02:24 printer_data/config/timelapse.cfg -> /home/lukics/moonraker-timelapse/klipper_macro/timelapse.cfg

and restore it to a different location I get this result:

-rw-r--r-- 1 root root  173 Feb  3 13:45 KlipperScreen.conf
-rw-r--r-- 1 root root   51 Feb  2 06:33 README.md
-rw-r--r-- 1 root root 1938 Jan 26 16:17 crowsnest.conf
drwxr-xr-x 2 root root 4096 Feb  4 16:27 macros
lrwxrwxrwx 1 root root   41 Feb  4 18:31 mainsail.cfg -> /home/lukics/mainsail-config/mainsail.cfg
-rw-r--r-- 1 root root 2466 Jan 26 16:19 moonraker.conf
-r--r--r-- 1 root root 7733 Feb  4 14:58 printer.cfg
drwxr-xr-x 2 root root 4096 Feb  1 22:44 printerconfig
-rw-r--r-- 1 root root  770 Dec 31 15:34 sonar.conf
-r--r--r-- 1 root root    0 Feb  4 18:10 test.cfg
-r--r--r-- 1 root root    0 Feb  4 18:20 test2.cfg
-r--r--r-- 1 root root    0 Feb  4 18:28 test3.cfg
lrwxrwxrwx 1 root root   60 Feb  4 18:31 timelapse.cfg -> /home/lukics/moonraker-timelapse/klipper_macro/timelapse.cfg

I have set options:

I would expect that the restored files have the same ownership as the source files.
Am I missing the point?

Xavron · February 6, 2023, 7:31pm

Looks like the new owner is root? Maybe its how you’re running Duplicati?

gpatel-fr · February 6, 2023, 7:37pm

Hello

I have not the restore-permissions checked either in the global options or in any job, yet owner, group and attributes are restored. As I don’t use Docker (I use vanilla Ubuntu), I’d say that the problem may come from that. Maybe the container don’t have enough rights to change this on a mounted volume ? (I am no expert in Docker).

Xavron · February 6, 2023, 10:57pm

I wonder if Duplicati just goes and does it anyway with different values when the environment doesn’t agree with the settings there. I’m not going to look at the code for this as it might very well be all over the place but you do raise an interesting question.

I’d expect if Duplicati’s settings are set to restore perms and it cannot then it should probably error out in some way. That should probably be the case even with what I said too. But, it didn’t.

Edit. I looked it up lol. Looks like it may be at the following: duplicati/SystemIOLinux.cs at beaf03562fdcf4425e962085bdf7175d6a465f49 · duplicati/duplicati · GitHub

In theory it could be throwing an error there and then something else would happen. If that’s the place then its one possibility as to why its not obeying the setting. I do that too at times with not marking probable bugs while coding even when I know at the time lol. But, Duplicati does have a lot of that.

Looks like a possible having no necessary perm on user object could cause it to fail there.

Other possibility here is if the data doesn’t contain the keys then this too would lead to it.

Ah. So the getUserID and other stuff here is in another class here: duplicati/File.cs at 666b2281032460254839fdc3b6e1055fdf7ce1db · duplicati/duplicati · GitHub

Looks like mono is involved too. That could also cause an issue.

And indeed, in mono it throws exceptions: mono/UnixUserInfo.cs at main · mono/mono · GitHub

gpatel-fr · February 6, 2023, 11:29pm

Probably should emit a warning in the log:

github.com

duplicati/duplicati/blob/666b2281032460254839fdc3b6e1055fdf7ce1db/Duplicati/Library/Main/Operation/RestoreHandler.cs#L300


      
                          if (!options.Dryrun && !SystemIO.IO_OS.DirectoryExists(folderpath))

                          {

                              Logging.Log.WriteWarningMessage(LOGTAG, "CreateMissingFolder", null, "Creating missing folder {0} for target {1}", folderpath, targetpath);

                              SystemIO.IO_OS.DirectoryCreate(folderpath);

                          }

          

                          ApplyMetadata(targetpath, metainfo.Value, options.RestorePermissions, options.RestoreSymlinkMetadata, options.Dryrun);

                      }

                      catch (Exception ex)

                      {

                          Logging.Log.WriteWarningMessage(LOGTAG, "MetadataWriteFailed", ex, "Failed to apply metadata to file: \"{0}\", message: {1}", targetpath, ex.Message);

                          if (options.UnittestMode)

                              throw;

                      }

                  }

              }

          }

          

          private void DoRun(LocalDatabase dbparent, Library.Utility.IFilter filter, RestoreResults result)

          {

              //In this case, we check that the remote storage fits with the database.

Xavron · February 6, 2023, 11:30pm

Ah yes. If it fails to write though. Not if it gets it wrong.

Idk. There’s too many possible problem points. Even mono itself could be a problem.

Might be safe to assume something with Docker is throwing it off?

I’m done here though. If an error shows up in the log or not might explain something. But, there’s not much else without debugging the code.

ts678 · February 7, 2023, 2:30am

A Docker did this in 2020, but I don’t think an issue was opened on it:

I don’t run Docker, so get lost quite easily in it. Whose Docker is this, and what is the PUID PGID plan?
Would a test as in the above-cited topic be possible? And what are these chown rights. Try them here?

Running on non-Docker Linux as a user, checking the restore-permissions box gives a warning where it would otherwise do a genuine change of user when running as root. That’s Linux – only root can do that unless root is also limited. High security systems have ways to trim root down from its usual high power.

Maybe some fiddling around inside Docker in a shell could test if a chown like Duplicati would do works.

Or if a vanilla (meaning no SELinux, AppArmor, etc.) non-Docker Linux is available, you could test that.

I’m kind of bothered by the differences in the permissions scrambling. Some went to R from RW on user, some didn’t. I wonder if it’s consistent or variable from run to run? For a small backup, it’s possible to get permissions from the dblock .zip files by hand. For tiny enough test backup, it’s just a couple of blocks.

Example of actual metadata block stored as a file (its name is the Base64 of its hash) inside one dblock

{“unix:uid-gid-perm”:“2000-2000-511”,“CoreAttributes”:“Normal”,“CoreLastWritetime”:“638112452291582755”,“CoreCreatetime”:“638112452291582755”}

is one where I forced file to UID 2000 which does not exist. I also wanted to test where it can’t find name, because inside a container the outside-the-container names might not be available, e.g. in /etc/passwd.

In a more normal situation, one would also have "unix:owner-name" and "unix:group-name" present.

drwtsn32 · February 7, 2023, 5:01am

I am thinking that may be the issue. Things might work better if Duplicati runs with root privileges.

lukics · March 8, 2023, 6:08am

You are right, I am running with a non-root user, actually, the user who owns the source files.

version: '3'
services:
  duplicati:
    image: duplicati/duplicati
    container_name: duplicati
    environment:
      - PUID=9001
      - PGID=9001
      - TZ=Europe/London
      - CLI_ARGS= #optional
    volumes:
      - /datavol/duplicati:/config
      - /media/lukics/backup-pro/duplicati:/backups
      - /datavol:/source/datavol
      - /home/lukics:/source/home
    ports:
      - 8200:8200
    restart: unless-stopped