Ticket 12556

Summary:	Reserve GPUs
Product:	Slurm	Reporter:	Amit Kumar <ahkumar>
Component:	Configuration	Assignee:	Scott Hilton <scott>
Status:	RESOLVED INFOGIVEN	QA Contact:
Severity:	3 - Medium Impact
Priority:	---
Version:	20.11.7
Hardware:	Linux
OS:	Linux
Site:	SMU	Alineos Sites:	---
Atos/Eviden Sites:	---	Confidential Site:	---
Coreweave sites:	---	Cray Sites:	---
DS9 clusters:	---	HPCnow Sites:	---
HPE Sites:	---	IBM Sites:	---
NOAA SIte:	---	OCF Sites:	---
Recursion Pharma Sites:	---	SFW Sites:	---
SNIC sites:	---	Linux Distro:	---
Machine Name:		CLE Version:
Version Fixed:		Target Release:	---
DevPrio:	---	Emory-Cloud Sites:	---

Description Amit Kumar 2021-09-27 14:55:36 MDT

Dear SchedMD,

I need to reserve 2 GPU out of 8 GPUs on a node. In other words need to reserve at total of 4 GPUs from 2 nodes and a handful of CPU cores. I looked at reservation with slurm, it appears it takes tres with cpu flags for reservation, wondering if I can use gpu flags to reserve only 2 GPUs per node. I was not able to use gpu glag hence the question. 

I need quick help on this as I need it for a workshop. Any help or tips is greatly appreciated. I can reserve the entire node but this nodes had too many GPUs and we don't want to waste them for a few hour workshop as there are workloads that have been waiting for days. 

Thank you!!

Comment 1 Scott Hilton 2021-09-27 15:24:51 MDT

Amit, 

Reservations currently cannot reserve gpus directly. To prevent other jobs from using the gpus with a reservation you will have to reserve the entire node.

I can't think of any good workarounds for your situation either.

-Scott

Comment 2 Amit Kumar 2021-09-29 12:59:48 MDT

Hi Scott thank you for your quick response!!