Ticket 2008

Summary:	Make job submission denied if memory requested > MemSpecLimit
Product:	Slurm	Reporter:	Akmal Madzlan <akmalm>
Component:	slurmctld	Assignee:	David Bigagli <david>
Status:	RESOLVED INFOGIVEN	QA Contact:
Severity:	5 - Enhancement
Priority:	---	CC:	brian, da
Version:	14.11.9
Hardware:	Linux
OS:	Linux
Site:	DownUnder GeoSolutions	Alineos Sites:	---
Atos/Eviden Sites:	---	Confidential Site:	---
Coreweave sites:	---	Cray Sites:	---
DS9 clusters:	---	HPCnow Sites:	---
HPE Sites:	---	IBM Sites:	---
NOAA SIte:	---	OCF Sites:	---
Recursion Pharma Sites:	---	SFW Sites:	---
SNIC sites:	---	Linux Distro:	---
Machine Name:		CLE Version:
Version Fixed:		Target Release:	---
DevPrio:	---	Emory-Cloud Sites:	---

Description Akmal Madzlan 2015-10-04 21:30:44 MDT

I have a node with MemSpecLimit=8000 and when submitting a job that request 15258, the job get submitted and run on that node

I think it doesnt really make sense if we allow job that request more memory than MemSpecLimit if we're going to kill them earlier.

I know MaxMemPerNode can be used to achieve this but I would like to set it per node instead of per partition

[akmalm@kud13 ~]$ scontrol show job 3259009
JobId=3259009 JobName=test
   UserId=akmalm(1419) GroupId=prod(2000)
   Priority=100 Nice=0 Account=(null) QOS=normal
   JobState=RUNNING Reason=None Dependency=(null)
   Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
   RunTime=00:00:10 TimeLimit=01:00:00 TimeMin=N/A
   SubmitTime=2015-10-02T09:57:01 EligibleTime=2015-10-02T09:57:01
   StartTime=2015-10-02T09:57:05 EndTime=2015-10-02T10:57:05
   PreemptTime=None SuspendTime=None SecsPreSuspend=0
   Partition=kud13 AllocNode:Sid=kud13:1843
   ReqNodeList=(null) ExcNodeList=(null)
   NodeList=kud13
   BatchHost=kud13
   NumNodes=1 NumCPUs=8 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
   Socks/Node=* NtasksPerN:B:S:C=1:0:*:* CoreSpec=*
   MinCPUsNode=1 MinMemoryNode=15258M MinTmpDiskNode=0
   Features=(null) Gres=(null) Reservation=(null)
   Shared=0 Contiguous=0 Licenses=(null) Network=(null)
   Command=/d/home/akmalm/rj.test.G9oHo8
   WorkDir=/d/home/akmalm
   Comment=/d/home/akmalm/test.job
   StdErr=/d/home/akmalm/logs/test.o3259009
   StdIn=/dev/null
   StdOut=/d/home/akmalm/logs/test.o3259009

[akmalm@kud13 ~]$ scontrol show node kud13
NodeName=kud13 Arch=x86_64 CoresPerSocket=2
   CPUAlloc=0 CPUErr=0 CPUTot=8 CPULoad=0.18 Features=localdisk
   Gres=(null)
   NodeAddr=kud13 NodeHostName=kud13 Version=14.11
   OS=Linux RealMemory=15947 AllocMem=0 Sockets=2 Boards=1
   MemSpecLimit=8000
   State=IDLE ThreadsPerCore=2 TmpDisk=674393 Weight=1
   BootTime=2015-09-07T09:56:22 SlurmdStartTime=2015-10-02T09:43:05
   CurrentWatts=0 LowestJoules=0 ConsumedJoules=0
   ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s

Comment 1 David Bigagli 2015-10-04 23:17:55 MDT

Hi,
   do you really mean MemSpecLimit? That is a limit for compute node daemons
the slurmd and slurmstepd.

David

Comment 2 Akmal Madzlan 2015-10-04 23:28:03 MDT

Yeah, I'm talking about MemSpecLimit.
When MemSpecLimit=8000, job that use more than 8000 will be killed right?

So I dont see the point of allowing the submission of job that request memory more than that

Comment 3 David Bigagli 2015-10-05 00:58:01 MDT

This is the documentation about the parameter. It is not a job limit.

MemSpecLimit
              Limit  on  combined real memory allocation for compute node dae-
              mons (slurmd, slurmstepd), in megabytes.

More informatioon can be found here: http://slurm.schedmd.com/core_spec.html.

David

Comment 4 Akmal Madzlan 2015-10-05 15:38:07 MDT

Right.
So can I make this a feature request?

Comment 5 Moe Jette 2015-10-05 15:59:52 MDT

(In reply to Akmal Madzlan from comment #4)
> Right.
> So can I make this a feature request?

It is already in Slum v15.08. You will need to use the MaxTRESPerJob parameter for Memory. More information here:

http://slurm.schedmd.com/SLUG15/TRES.pdf
http://slurm.schedmd.com/resource_limits.html
http://slurm.schedmd.com/sacctmgr.html

Comment 6 Akmal Madzlan 2015-10-05 21:44:15 MDT

Alright.
Thanks Moe