Ticket 8197

Summary:	DenyOnLimit not working
Product:	Slurm	Reporter:	Ilya Draigor <ilyadr>
Component:	Limits	Assignee:	Jess <jess>
Status:	OPEN ---	QA Contact:
Severity:	6 - No support contract
Priority:	---
Version:	19.05.4
Hardware:	Linux
OS:	Linux
Site:	-Other-	Alineos Sites:	---
Atos/Eviden Sites:	---	Confidential Site:	---
Coreweave sites:	---	Cray Sites:	---
DS9 clusters:	---	HPCnow Sites:	---
HPE Sites:	---	IBM Sites:	---
NOAA SIte:	---	NoveTech Sites:	---
Nvidia HWinf-CS Sites:	---	OCF Sites:	---
Recursion Pharma Sites:	---	SFW Sites:	---
SNIC sites:	---	Tzag Elita Sites:	---
Linux Distro:	---	Machine Name:
CLE Version:		Version Fixed:
Target Release:	---	DevPrio:	---
Emory-Cloud Sites:	---

Description Ilya Draigor 2019-12-08 05:32:20 MST

Hi,

Is this a bug or i'm doining something wrong?

i need to limit QOS for number of gpu uses, for this purpose i set MaxTRESPerAccount=gres/gpu=1 (also tried with GrpTRES) and Flags=DenyOnLimit

when i run sbatch for first job (with --gres=gpu:1) all fine and job is running, but when i run a second job it's marked in queue as (MaxGRESPerAccount) but not rejecting on submission, so DenyOnLimit not working

checked on slurm 19.05.3-2 and 19.05.4-1

My second question, what is the best solution to guarantee gpu cards to users who bought them ?
Let's say, user bought 2 gpu cards for cluster, and i need to guarantee that this user always got immediately access to this cards even if another job must be preempted
as i see, the solution for this is to give user a qos with preemption rights and limit maximum usage for 2 gpus, otherwise he can preempt all running jobs on whole cluster.

Hope for help with this issue

Thanks.

Regards,
Ilya

Comment 1 Jacob Jenson 2019-12-09 09:55:06 MST

Ilya,

In reference to both questions, the functionality you want can be accomplished with Slurm. You need to make configuration changes and changes in how jobs are submitted. 

SchedMD has a commercial support team that can tell you the specific changes you will need to make. However, before you can engage with the support team your site will need to purchase a support contract. Can you please tell me who we should talk with at your site regarding purchasing a Slurm support contract? 

Thank you,
Jacob

Comment 2 Ilya Draigor 2019-12-09 10:09:09 MST

Hi Jacob,
Thanks for reply
I was installed and configured slurm, so you can talk with me

Thanks