Ticket 4455

Created attachment 5646 [details]
Job log from GPU test run

My job log doesn't cover 2 jobs running at the same time, it presents a single job being able to access all 4 GPUs, keeps it clean and simple.
And the job log submitted is for the slurmd running the full release of 17.11.0

Adam,

I reproduced the problem, but I actually had the same problem on 17.02 as well as 17.11. I'm looking for the source of the problem and will be working on a fix now.

Thanks Marshall, was that 17.02.10 you were testing on, I don't believe I noticed the problem on the other 17.02.[4,5,9]

I learned something else. If ConstrainDevices=yes in your cgroup.conf, then the job only ever sees the GPUs it was allocated. Since each job was only allocated 2 GPUs, the you see indicies 0,1 for CUDA_VISIBLE_DEVICES on each job. But I think they should still be the 4 different GPUs, not the same 2.

If I set ConstrainDevices=no in my cgroup.conf, then the job sees all GPUs but seems to just use the ones it was allocated.

marshall@byu:~/slurm/17.11/byu$ salloc --gres=gpu:2
salloc: Granted job allocation 5448
marshall@byu:~/slurm/17.11/byu$ srun env | grep -i cuda
CUDA_VISIBLE_DEVICES=0,1

marshall@byu:~/slurm/17.11/byu/slurm$ salloc --gres=gpu:2
salloc: Granted job allocation 5449
marshall@byu:~/slurm/17.11/byu/slurm$ srun env | grep -i cuda
CUDA_VISIBLE_DEVICES=2,3




> running the CUDA bandwidthTest sample suite
> runs against the first 2 GPUs in this case on both jobs, the second job will
> fail if the first one is using nvidia[0,1] already.

Can you double check this for me that this is a Slurm issue, not something else? Can you try it with ConstrainDevices=no as well as ConstrainDevices=yes? It should work both times (assuming there aren't any issues outside of Slurm)


I was running 17.02.10 and 17.11.0. However, I just tested on 17.02.9 and got the same behavior.

Sorry, I should have mentioned it at the start, we are using ConstrainDevices=yes in production and the initial testing that I displayed.

I've tested with ConstrainDevices=no and it does set CUDA_VISIBLE_DEVICES appropriately in 17.11.0 (second job gets 2,3)  

When ConstrainDevices=yes then CUDA_VISIBLE_DEVICES should always start at 0, and then if 2 devices are chosen it will be 0,1

In a correctly working setup, with ConstrainDevices=yes, if the job gets constrained to /dev/nvidia2 and /dev/nvidia3 then CUDA_VISIBLE_DEVICES will be 0,1 which is correct and it will cause them to operate only on those 2 /dev/nvidia devices.

What I am seeing is in 17.11.0 with ConstrainDevices=yes is that CUDA_VISIBLE_DEVICES is correct in saying 0,1 but because the node is failing to constrain them to their allocated /dev/nvidia devices, they end up seeing all 4 and 0,1 equals /dev/nvidia0 /dev/nvidia1 in that case.... which means any job on that node will at least attempt to use /dev/nvidia0 and run into a conflict.

I am unable to run into this issue using 17.02.[4,5,9] with ConstrainDevices=yes

While we did upgrade nvidia driver versions at the same time as we upgraded Slurm versions, I'm not convinced it could be the cause since i'm unable to reproduce the breakage in the older Slurm version.

I think the easiest way to prove that it's not Constraining properly is to just do a job that requests 1 GPU and runs $(nvidia-smi), if it shows more than 1 GPU on a multigpu node, then it's broken.

You're right, it was broken. The cgroup wasn't correctly adding devices to devices.deny and devices.allow. It was correct in 17.02, but broke in 17.11. A patch will be forthcoming.

Thanks Marshall.  I should post the other 2 bugs that I know about for 17.11.0 then ;)

This has been fixed in commit 434acb17c8526b.

Looks like I jumped the gun on closing this bug. There are still problems, though the previously mentioned patch did fix a big one.

Devices aren't being denied if the user doesn't request any GRES.

Parsing the cgroup.conf file isn't ignoring comments and blank lines.

There may be other problems, too.

Reopening the bug.

Devices are properly being denied, see commit 0ed03cda5bcf4. Missing debug statement is fixed in commit ee68721350dc46. cgroup.conf does ignore comments, blanks lines, etc. - it just tells you about it when you have debug3 on, which is why I was mistaken.

Closing as resolved/fixed.

I forgot to actually close the bug in the previous comment... Closing as resolved/fixed.

*** Ticket 4518 has been marked as a duplicate of this ticket. ***

Quick update note.  Thanks for the quick fix on the bug guys.  We've brought it active at 2 sites and it's working smoothly.  We built from the top of the 17.11 branch as of Dec 11, 2017.