I recently tried Sharding (gres:shard) (https://slurm.schedmd.com/gres.html#Sharding) to have GPUs shared across multiple jobs -- which gres:mps does not support. This works well as expected, but there is one big problem: sharding-allocated GPU resources seem to be allocated and managed independent of conventional gres:GPU resources. When one (physical) GPU is allocated to one or multiple jobs with --gres=shard:X, then this GPU should NOT be allocated or available for "gres:gpu" as "gres:gpu" works in an exclusive way. Let's say two shard (--gres shard:4, shard:6) jobs occupy 4GB + 6 GB = 10GB out of 12GB VRAM of a GPU (Shard count=12), and we want to assign another job with --gres gpu:1. Then this gpu:1 job might be allocated to the GPU which are already occupied by Sharding GPUs, resulting in a failure (OOM). Therefore, gres:gpu should be aware of gres:shard and vice versa. Example slurm.conf (excerpt): ``` AccountingStorageTRES=gres/gpu,gres/shard GresTypes=gpu,shard SelectType=select/cons_tres NodeName=node1 Gres=gpu:4,shard:48 Sockets=2 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=96664 State=UNKNOWN ``` gres.conf: ``` NodeName=node1 Name=gpu File=/dev/nvidia0 NodeName=node1 Name=shard Count=12 File=/dev/nvidia0 NodeName=node1 Name=gpu File=/dev/nvidia1 NodeName=node1 Name=shard Count=12 File=/dev/nvidia1 NodeName=node1 Name=gpu File=/dev/nvidia2 NodeName=node1 Name=shard Count=12 File=/dev/nvidia2 NodeName=node1 Name=gpu File=/dev/nvidia3 NodeName=node1 Name=shard Count=12 File=/dev/nvidia3 ``` Allocation: ``` $ srun -p legacy --oversubscribe -N1 -n1 -c1 --gres shard:5 --mem=1 --time=23:59:59 -w node1 ... # JOB 1 $ srun -p legacy --oversubscribe -N1 -n1 -c1 --gres shard:5 --mem=1 --time=23:59:59 -w node1 ... # JOB 2 # JOB 1 and 2 will be allocated on GPU 0 $ srun -p legacy --oversubscribe -N1 -n1 -c1 --gres gpu:1 --mem=1 --time=23:59:59 -w node1 ... # JOB 3 # JOB 3 should be allocated on GPU 1, 2, or 3, but it gets allocated on GPU 0 ```
The documentation says > Note the same GPU can be allocated either as a GPU type of GRES or as a shard type of GRES, but not both. In other words, once a GPU has been allocated as a gres/gpu resource it will not be available as a gres/shard. Likewise, once a GPU has been allocated as a gres/shard resource it will not be available as a gres/gpu. and the behavior is not like this description, so I guess this is simply a bug.
I found that the `AutoDetect=nvml` line in gres.conf makes a difference -- with this line added, everything works as expected; a gres/gpu allocated GPU won't be available to gres/shard and vice versa. Not sure why the resource allocation works unexpectedly rather than an error, or what exactly happens under the hood without this config option (there is no warning/error), but this can be still confusing.
I am having this issue as well. It's possible to use up all the available shards and then consume the GPUs as a separate resource, leading to an overuse of resources. Slurm 22.05.3 on Rocky 8.8 Kind regards, Michael
I have re-visited this issue on 24.11.1 and it now seems to have been resolved. I tried to submit a --gpus=1 and --gres=shard:1 job at the same time and they would not run together. I think this is the correct behaviour. Kind regards, Michael
