Ticket 12556 - Reserve GPUs
Summary: Reserve GPUs
Status: RESOLVED INFOGIVEN
Alias: None
Product: Slurm
Classification: Unclassified
Component: Configuration (show other tickets)
Version: 20.11.7
Hardware: Linux Linux
: 3 - Medium Impact
Assignee: Scott Hilton
QA Contact:
URL:
Depends on:
Blocks:
 
Reported: 2021-09-27 14:55 MDT by Amit Kumar
Modified: 2021-09-29 12:59 MDT (History)
0 users

See Also:
Site: SMU
Slinky Site: ---
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
Google sites: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
NoveTech Sites: ---
Nvidia HWinf-CS Sites: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Tzag Elita Sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed:
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this ticket.
Description Amit Kumar 2021-09-27 14:55:36 MDT 
Dear SchedMD,

I need to reserve 2 GPU out of 8 GPUs on a node. In other words need to reserve at total of 4 GPUs from 2 nodes and a handful of CPU cores. I looked at reservation with slurm, it appears it takes tres with cpu flags for reservation, wondering if I can use gpu flags to reserve only 2 GPUs per node. I was not able to use gpu glag hence the question. 

I need quick help on this as I need it for a workshop. Any help or tips is greatly appreciated. I can reserve the entire node but this nodes had too many GPUs and we don't want to waste them for a few hour workshop as there are workloads that have been waiting for days. 

Thank you!!
Comment 1 Scott Hilton 2021-09-27 15:24:51 MDT 
Amit, 

Reservations currently cannot reserve gpus directly. To prevent other jobs from using the gpus with a reservation you will have to reserve the entire node.

I can't think of any good workarounds for your situation either.

-Scott
Comment 2 Amit Kumar 2021-09-29 12:59:48 MDT 
Hi Scott thank you for your quick response!!