Ticket 9346

Summary: GPU
Product: Slurm Reporter: Marc Caubet Serrabou <marc.caubet>
Component: GPUAssignee: Scott Hilton <scott>
Status: RESOLVED INFOGIVEN QA Contact:
Severity: 4 - Minor Issue    
Priority: --- CC: cinek, spencer.bliven
Version: 19.05.7   
Hardware: Linux   
OS: Linux   
See Also: https://bugs.schedmd.com/show_bug.cgi?id=10569
Site: Paul Scherrer Slinky Site: ---
Alineos Sites: --- Atos/Eviden Sites: ---
Confidential Site: --- Coreweave sites: ---
Cray Sites: --- DS9 clusters: ---
Google sites: --- HPCnow Sites: ---
HPE Sites: --- IBM Sites: ---
NOAA SIte: --- NoveTech Sites: ---
Nvidia HWinf-CS Sites: --- OCF Sites: ---
Recursion Pharma Sites: --- SFW Sites: ---
SNIC sites: --- Tzag Elita Sites: ---
Linux Distro: --- Machine Name:
CLE Version: Version Fixed:
Target Release: --- DevPrio: ---
Emory-Cloud Sites: ---

Description Marc Caubet Serrabou 2020-07-07 07:12:36 MDT 
Hi,

we have 3 different models of GPU, GTX1080 (8GB mem), GTX1080Ti and RTX2080Ti (11GB mem). GRES is defined as follows:

NodeName=merlin-g-001 Name=gpu Type=GTX1080 File=/dev/nvidia[0,1] Cores=0-15
NodeName=merlin-g-00[2-5] Name=gpu Type=GTX1080 File=/dev/nvidia[0,1] Cores=0-9
NodeName=merlin-g-00[2-5] Name=gpu Type=GTX1080 File=/dev/nvidia[2,3] Cores=10-19
NodeName=merlin-g-00[6-9] Name=gpu Type=GTX1080Ti File=/dev/nvidia[0,1] Cores=0-9
NodeName=merlin-g-00[6-9] Name=gpu Type=GTX1080Ti File=/dev/nvidia[2,3] Cores=10-19
NodeName=merlin-g-01[0-3] Name=gpu Type=RTX2080Ti File=/dev/nvidia[0,1] Cores=0-9
NodeName=merlin-g-01[0-3] Name=gpu Type=RTX2080Ti File=/dev/nvidia[2,3] Cores=10-19

and in slurm.conf:

AccountingStorageTRES=gres/gpu,gres/gpu:GTX1080,gres/gpu:GTX1080Ti,gres/gpu:RTX2080Ti

However, we have some jobs that would require 11GB mem based GPUs (it is, to choose any combination of GTX1080Ti and RXT2080Ti), but he above config seems not allowing that.

Which is the best way for specifying and configuring this? Would this require a new GRES type or Feature needed? Any recommendations?

Thanks a lot,
Marc
Comment 2 Scott Hilton 2020-07-07 10:20:53 MDT 
Marc,

I think the best solution is to use a feature. 

You could give the proper nodes a feature called 11GB_GPU.

Then the user will request 4 gpus, for example, by calling something like:
>sbatch -C 11GB_GPU --gres=gpu:4 ./myjob.sh

Let me know if you have any follow up questions.

Good luck,

Scott
Comment 5 Marc Caubet Serrabou 2020-07-08 01:51:26 MDT 
Hi Scott,

thanks for the fast answer. Ok, I was evaluating this option, which is ok for us.

Just one last question related to this, --mem-per-gpu I guess only applies to physical RAM memory, not to GPU memory, correct? 

We are still using cons_res instead of cons_tres (we probably will switch to the latter as soon as we migrate to Slurm v20), would be nice to define a way for selecting GPUs according to the memory GPU mem available. Is this for curiosity in the roadmap? 

Cheers,
Marc
Comment 6 Scott Hilton 2020-07-08 12:12:14 MDT 
Marc, 

You are correct, --mem-per-gpu looks at node memory not gpu memory.

There are currently no plans to add anything like this to slurm.

-Scott
Comment 7 Marc Caubet Serrabou 2020-07-09 01:24:20 MDT 
Hi,

thanks a lot for the answer. Ok, a parameter defining GPU memory would be a good improvement. In any case, thanks a lot for your help, we will proceed with Features for defining that. Ticket can be closed.

Best regards,
Marc
Comment 8 Scott Hilton 2020-07-09 08:27:37 MDT 
Ok, good luck
Comment 9 Spencer Bliven 2021-01-06 03:23:27 MST 
We're trying to implement this now.

One issue with features is that they are binary, so if we add additional GPU types in the future then users will have to manually add those features to the constraints. It seems like doing this with GRES would be better and would allow users to just specify minimum requirements instead of a list of multiple features (--gres=gpumem:11G instead of --constraint="11G_GPU|16G_GPU").

However, we ran into issues when implementing the GRES approach, as described in #10569.

I'll leave this closed for now, but I wanted to reference the other ticket here.