Ticket 2130

Hi,

We're using the slurm_pam_adopt PAM module in conjunction with the cgroup devices subsystem, in order to restrict user access to GPUs. It works mostly fine, except for a corner case I can't seem to figure out: when a user SSH to a node, her shell doesn't inherit the cgroup devices settings, but is instead denied access to all devices.

So, it's basically:
1. srun --gres gpu:2 --pty bash -> can see 2 GPUs
2. ssh to the node with pam_slurm_adopt -> can't see any GPU at all


Our config is as follows:
JobAcctGatherType       = jobacct_gather/cgroup
ProctrackType           = proctrack/cgroup
PrologFlags             = Alloc,Contain
TaskPlugin              = task/cgroup

cgroup.conf:
ConstrainDevices=yes
AllowedDevicesFile=/etc/slurm/cgroup_allowed_devices_file.conf


When I run this job, I correctly get access to only 2 GPUs:

$ srun -p test --gres gpu:2 --pty bash
xs-0057 $ nvidia-smi -L
GPU 0: Tesla K80 (UUID: GPU-198795b7-0588-3272-ee67-7b395851387f)
GPU 1: Tesla K80 (UUID: GPU-68e053ec-d3d2-92c5-5e19-3a12d2444d4a)
xs-0057 $

The cgroups eem to be set correctly, as the logs show, first for step_extern, then for step_0:

slurmd[5840]: launch task 5736.0 request from 215845.2709@10.10.0.21 (port 10681)
slurmstepd[6248]: task/cgroup: /slurm/uid_215845/job_5736: alloc=12000MB mem.limit=12000MB memsw.limit=12000MB
slurmstepd[6248]: task/cgroup: /slurm/uid_215845/job_5736/step_extern: alloc=12000MB mem.limit=12000MB memsw.limit=12000MB
slurmstepd[6248]: task/cgroup: manage devices jor job '5736'
[...]
slurmstepd[6248]: Allowing access to device c 195:0 rwm
slurmstepd[6248]: Allowing access to device c 195:1 rwm
slurmstepd[6248]: Not allowing access to device c 195:2 rwm
slurmstepd[6248]: Not allowing access to device c 195:3 rwm
slurmstepd[6248]: Not allowing access to device c 195:4 rwm
slurmstepd[6248]: Not allowing access to device c 195:5 rwm
slurmstepd[6248]: Not allowing access to device c 195:6 rwm
slurmstepd[6248]: Not allowing access to device c 195:7 rwm
slurmstepd[6248]: Not allowing access to device c 195:8 rwm
[...]
slurmstepd[6254]: task/cgroup: /slurm/uid_215845/job_5736: alloc=12000MB mem.limit=12000MB memsw.limit=12000MB
slurmstepd[6254]: task/cgroup: /slurm/uid_215845/job_5736/step_0: alloc=12000MB mem.limit=12000MB memsw.limit=12000MB
slurmstepd[6254]: task/cgroup: manage devices jor job '5736'
[...]
slurmstepd[6254]: Allowing access to device c 195:0 rwm
slurmstepd[6254]: Allowing access to device c 195:1 rwm
slurmstepd[6254]: Not allowing access to device c 195:2 rwm
slurmstepd[6254]: Not allowing access to device c 195:3 rwm
slurmstepd[6254]: Not allowing access to device c 195:4 rwm
slurmstepd[6254]: Not allowing access to device c 195:5 rwm
slurmstepd[6254]: Not allowing access to device c 195:6 rwm
slurmstepd[6254]: Not allowing access to device c 195:7 rwm
slurmstepd[6254]: Not allowing access to device c 195:8 rwm
[...]


Then, when I ssh to the node, the PID of the new shell is correctly added to the step_extern's process list, yet I can't seem to list the GPUs:

$ ssh xs-0057
xs-0057 $ echo $$
7202
xs-0057 $ cat /cgroup/devices/slurm/uid_215845/job_5736/step_extern/cgroup.procs
6248
6258
7197
7201
7202
7418
xs-0057 $ nvidia-smi -L
No devices found.


Since I was not entirely sure if it was a Slurm issue or more a cgroup issue, I tried to do this by hand:

1. create a new cgroup:
xs-0057 # mkdir /cgroup/devices/test/

2. ssh as a user to the node:
$ ssh xs-0057
xs-0057 $ echo $$
8216

3. configure the cgroup (195:* devices are the /dev/nvidia* devices, 195:255 is /dev/nvidiactl)
xs-0057 # echo 8216 > /cgroup/devices/test/cgroup.procs
xs-0057 # echo a > /cgroup/devices/test/devices.deny
xs-0057 # echo "c 195:0 rwm" > /cgroup/devices/test/devices.allow
xs-0057 # echo "c 195:1 rwm" > /cgroup/devices/test/devices.allow
xs-0057 # echo "c 195:255 rwm" > /cgroup/devices/test/devices.allow
xs-0057 # cat /cgroup/devices/test/devices.list
c 195:0 rwm
c 195:1 rwm
c 195:255 rwm

4. check that access restriction works as a user:
xs-0057 $ nvidia-smi -L
GPU 0: Tesla K80 (UUID: GPU-198795b7-0588-3272-ee67-7b395851387f)
GPU 1: Tesla K80 (UUID: GPU-68e053ec-d3d2-92c5-5e19-3a12d2444d4a)

5. now SSH in a new shell, add its PID to the test cgroup, and see what the device access is:
$ ssh xs-0057 
xs-0057 $ echo $$
9094

as root in another session:
xs-0057 # echo 9094 > /cgroup/devices/test/cgroup.procs

back as the user:
xs-0057 $ echo $$
9094
xs-0057 $ nvidia-smi -L
GPU 0: Tesla K80 (UUID: GPU-198795b7-0588-3272-ee67-7b395851387f)
GPU 1: Tesla K80 (UUID: GPU-68e053ec-d3d2-92c5-5e19-3a12d2444d4a)

So the new shell correctly inherits from the device access restriction when its PID is added to the cgroup's list of processes.

That makes me think that there is something different in the was Slurm initiates and or sets the cgroups up.

Any insight would be much appreciated.

Thanks,
-- 
Kilian


PS: Something that may be related (or not), is that the devices.list file in the cgroups set by Slurm always only contains "a *:* rwm", whatever the access restrictions are:

xs-0057 # tree /cgroup/devices/slurm/
/cgroup/devices/slurm/
├── cgroup.event_control
├── cgroup.procs
├── devices.allow
├── devices.deny
├── devices.list
├── notify_on_release
├── tasks
└── uid_215845
    ├── cgroup.event_control
    ├── cgroup.procs
    ├── devices.allow
    ├── devices.deny
    ├── devices.list
    ├── job_5738
    │   ├── cgroup.event_control
    │   ├── cgroup.procs
    │   ├── devices.allow
    │   ├── devices.deny
    │   ├── devices.list
    │   ├── notify_on_release
    │   ├── step_0
    │   │   ├── cgroup.event_control
    │   │   ├── cgroup.procs
    │   │   ├── devices.allow
    │   │   ├── devices.deny
    │   │   ├── devices.list
    │   │   ├── notify_on_release
    │   │   └── tasks
    │   ├── step_extern
    │   │   ├── cgroup.event_control
    │   │   ├── cgroup.procs
    │   │   ├── devices.allow
    │   │   ├── devices.deny
    │   │   ├── devices.list
    │   │   ├── notify_on_release
    │   │   └── tasks
    │   └── tasks
    ├── notify_on_release
    └── tasks
# find /cgroup/devices/slurm/ -iname devices.list -exec cat {} \;
a *:* rwm
a *:* rwm
a *:* rwm
a *:* rwm
a *:* rwm

This is not the case when setting the cgroups manually, as shown above: the devices.list correctly contains the list of whitelisted devices.