Ticket 16484

Summary:	gres:shard GPUs are not exclusive with gres:gpu
Product:	Slurm	Reporter:	wookayin
Component:	GPU	Assignee:	Jacob Jenson <jacob>
Status:	OPEN ---	QA Contact:
Severity:	6 - No support contract
Priority:	---	CC:	michael
Version:	23.11.x
Hardware:	Linux
OS:	Linux
Site:	-Other-	Slinky Site:	---
Alineos Sites:	---	Atos/Eviden Sites:	---
Confidential Site:	---	Coreweave sites:	---
Cray Sites:	---	DS9 clusters:	---
Google sites:	---	HPCnow Sites:	---
HPE Sites:	---	IBM Sites:	---
NOAA SIte:	---	NoveTech Sites:	---
Nvidia HWinf-CS Sites:	---	OCF Sites:	---
Recursion Pharma Sites:	---	SFW Sites:	---
SNIC sites:	---	Tzag Elita Sites:	---
Linux Distro:	---	Machine Name:
CLE Version:		Version Fixed:
Target Release:	---	DevPrio:	---
Emory-Cloud Sites:	---

Description wookayin 2023-04-08 08:10:27 MDT

I recently tried Sharding (gres:shard) (https://slurm.schedmd.com/gres.html#Sharding) to have GPUs shared across multiple jobs -- which gres:mps does not support. This works well as expected, but there is one big problem: sharding-allocated GPU resources seem to be allocated and managed independent of conventional gres:GPU resources.

When one (physical) GPU is allocated to one or multiple jobs with --gres=shard:X, then this GPU should NOT be allocated or available for "gres:gpu" as "gres:gpu" works in an exclusive way. Let's say two shard (--gres shard:4, shard:6) jobs occupy 4GB + 6 GB = 10GB out of 12GB VRAM of a GPU (Shard count=12), and we want to assign another job with --gres gpu:1. Then this gpu:1 job might be allocated to the GPU which are already occupied by Sharding GPUs, resulting in a failure (OOM). Therefore, gres:gpu should be aware of gres:shard and vice versa. 

Example slurm.conf (excerpt):
```
AccountingStorageTRES=gres/gpu,gres/shard
GresTypes=gpu,shard
SelectType=select/cons_tres

NodeName=node1 Gres=gpu:4,shard:48 Sockets=2 CoresPerSocket=6 ThreadsPerCore=2 RealMemory=96664 State=UNKNOWN
```

gres.conf:
```
NodeName=node1 Name=gpu File=/dev/nvidia0
NodeName=node1 Name=shard Count=12 File=/dev/nvidia0
NodeName=node1 Name=gpu File=/dev/nvidia1
NodeName=node1 Name=shard Count=12 File=/dev/nvidia1
NodeName=node1 Name=gpu File=/dev/nvidia2
NodeName=node1 Name=shard Count=12 File=/dev/nvidia2
NodeName=node1 Name=gpu File=/dev/nvidia3
NodeName=node1 Name=shard Count=12 File=/dev/nvidia3
```

Allocation:
```
$ srun -p legacy --oversubscribe -N1 -n1 -c1 --gres shard:5 --mem=1 --time=23:59:59 -w node1 ...   # JOB 1
$ srun -p legacy --oversubscribe -N1 -n1 -c1 --gres shard:5 --mem=1 --time=23:59:59 -w node1 ...   # JOB 2

# JOB 1 and 2 will be allocated on GPU 0

$ srun -p legacy --oversubscribe -N1 -n1 -c1 --gres gpu:1 --mem=1 --time=23:59:59 -w node1 ...   # JOB 3

# JOB 3 should be allocated on GPU 1, 2, or 3, but it gets allocated on GPU 0
```

Comment 1 wookayin 2023-04-08 09:35:48 MDT

The documentation says

> Note the same GPU can be allocated either as a GPU type of GRES or as a shard type of GRES, but not both. In other words, once a GPU has been allocated as a gres/gpu resource it will not be available as a gres/shard. Likewise, once a GPU has been allocated as a gres/shard resource it will not be available as a gres/gpu.

and the behavior is not like this description, so I guess this is simply a bug.

Comment 2 wookayin 2023-04-08 09:50:33 MDT

I found that the `AutoDetect=nvml` line in gres.conf makes a difference -- with this line added, everything works as expected; a gres/gpu allocated GPU won't be available to gres/shard and vice versa. Not sure why the resource allocation works unexpectedly rather than an error, or what exactly happens under the hood without this config option (there is no warning/error), but this can be still confusing.

Comment 3 Michael Bell 2023-10-20 07:02:22 MDT

I am having this issue as well. It's possible to use up all the available shards and then consume the GPUs as a separate resource, leading to an overuse of resources.

Slurm 22.05.3 on Rocky 8.8

Kind regards,

Michael

Comment 4 Michael Bell 2025-06-05 06:52:44 MDT

I have re-visited this issue on 24.11.1 and it now seems to have been resolved. I tried to submit a --gpus=1 and --gres=shard:1 job at the same time and they would not run together. I think this is the correct behaviour. 

Kind regards,

Michael