Ticket 4518

Sorry for the barrage of submissions today, but as you probably guessed, we've upgraded to 17.11. And transition is not as smooth as we expected, unfortunately. :\

It looks like ContraintDevices doesn't work anymore in 17.11. Using the exact same (functional) config as we used in 17.02, we can't seem to make device access restrictions work.

For instance, when requesting a single GPU, I can see all the GPUs on the node:

$ srun --pty --gres gpu:1 -w sh-112-07 bash

[kilian@sh-112-07 ~]$ nvidia-smi -L
GPU 0: Tesla P100-PCIE-16GB (UUID: GPU-27db8534-9b2b-8b1a-5889-9c77c0c7be4e)
GPU 1: Tesla P100-PCIE-16GB (UUID: GPU-b84691c7-e6e4-33c1-f367-1831e42cf4c6)
GPU 2: Tesla P100-PCIE-16GB (UUID: GPU-0d9a859c-ce19-78f3-2f87-aade11d14bae)
GPU 3: Tesla P100-PCIE-16GB (UUID: GPU-59405c9d-6554-d857-1cbd-8b9487464684)


The devices cgroups seem to have been created properly:

[kilian@sh-112-07 ~]$ echo $SLURM_JOBID
4542325
[kilian@sh-112-07 ~]$ lscgroup | grep devices
devices:/
devices:/slurm
devices:/slurm/uid_215845/job_4542325
devices:/slurm/uid_215845/job_4542325/step_0
devices:/slurm/uid_215845/job_4542325/step_extern
[kilian@sh-112-07 ~]$ tree /sys/fs/cgroup/devices/slurm/uid_215845/job_4542065/
/sys/fs/cgroup/devices/slurm/uid_215845/job_4542065/
├── cgroup.clone_children
├── cgroup.event_control
├── cgroup.procs
├── devices.allow
├── devices.deny
├── devices.list
├── notify_on_release
├── step_extern
│   ├── cgroup.clone_children
│   ├── cgroup.event_control
│   ├── cgroup.procs
│   ├── devices.allow
│   ├── devices.deny
│   ├── devices.list
│   ├── notify_on_release
│   └── tasks
└── tasks


The GPUs are correctly detected on the node:

slurmd[109644]: Gres Name=gpu Type=(null) Count=2
slurmd[109644]: Gres Name=gpu Type=(null) Count=2
slurmd[109644]: gpu device number 0(/dev/nvidia0):c 195:0 rwm
slurmd[109644]: gpu device number 1(/dev/nvidia1):c 195:1 rwm
slurmd[109644]: gpu device number 2(/dev/nvidia2):c 195:2 rwm
slurmd[109644]: gpu device number 3(/dev/nvidia3):c 195:3 rwm


The cgroup config is loaded, including the devices part:

slurmd[109644]: debug:  Reading cgroup.conf file /etc/slurm/cgroup.conf
slurmd[109644]: debug:  CPUs:20 Boards:1 Sockets:2 CoresPerSocket:10 ThreadsPerCore:1
slurmd[109644]: debug:  Reading cgroup.conf file /etc/slurm/cgroup.conf
slurmd[109644]: debug:  task/cgroup: now constraining jobs allocated cores
slurmd[109644]: debug:  task/cgroup/memory: total:515705M allowed:100%(enforced), swap:0%(enforced), max:100%(515705M) max+swap:100%(1031410M) min:30M kmem:100%(515705M permissive) min:30M swappiness:18446744073709551614(set)
slurmd[109644]: debug:  task/cgroup: now constraining jobs allocated memory
slurmd[109644]: debug:  task/cgroup: now constraining jobs allocated devices
slurmd[109644]: debug:  task/cgroup: loaded



Yet, there's no mention of GRES devices when the job start:


slurmd[109653]: debug:  Checking credential with 448 bytes of sig data
slurmd[109653]: debug:  Reading cgroup.conf file /etc/slurm/cgroup.conf
slurmd[109653]: debug:  Calling /usr/sbin/slurmstepd spank prolog
slurmd[109653]: debug:  [job 4542325] attempting to run prolog [/etc/slurm/scripts/prolog.sh]
slurmd[109653]: _run_prolog: run job script took usec=38237
slurmd[109653]: _run_prolog: prolog with lock for job 4542325 ran for 0 seconds
slurmstepd[109874]: task/cgroup: /slurm/uid_215845/job_4542325: alloc=12800MB mem.limit=12800MB memsw.limit=12800MB
slurmstepd[109874]: task/cgroup: /slurm/uid_215845/job_4542325/step_extern: alloc=12800MB mem.limit=12800MB memsw.limit=12800MB
slurmd[109653]: launch task 4542325.0 request from 215845.32264@10.10.0.61 (port 54987)
slurmd[109653]: debug:  Checking credential with 492 bytes of sig data
slurmd[109653]: debug:  Waiting for job 4542325's prolog to complete
slurmd[109653]: debug:  Finished wait for job 4542325's prolog to complete
slurmstepd[109881]: task/cgroup: /slurm/uid_215845/job_4542325: alloc=12800MB mem.limit=12800MB memsw.limit=12800MB
slurmstepd[109881]: task/cgroup: /slurm/uid_215845/job_4542325/step_0: alloc=12800MB mem.limit=12800MB memsw.limit=12800MB
slurmstepd[109881]: in _window_manager


In previous versions, there was a part that said:

slurmstepd[6248]: Allowing access to device c 195:0 rwm
slurmstepd[6248]: Allowing access to device c 195:1 rwm
slurmstepd[6248]: Not allowing access to device c 195:2 rwm
slurmstepd[6248]: Not allowing access to device c 195:3 rwm

which doesn't appear in the logs anymore.


Is there any additional configuration required in 17.11 to use the devices cgroup constraints?

Thanks!
-- 
Kilian