Ticket 2994

Summary:	Max number of instances of 1 job array.
Product:	Slurm	Reporter:	paull
Component:	Scheduling	Assignee:	Moe Jette <jette>
Status:	RESOLVED CANNOTREPRODUCE	QA Contact:
Severity:	4 - Minor Issue
Priority:	---	CC:	alex
Version:	14.11.10
Hardware:	Linux
OS:	Linux
Site:	DownUnder GeoSolutions	Alineos Sites:	---
Atos/Eviden Sites:	---	Confidential Site:	---
Coreweave sites:	---	Cray Sites:	---
DS9 clusters:	---	HPCnow Sites:	---
HPE Sites:	---	IBM Sites:	---
NOAA SIte:	---	NoveTech Sites:	---
Nvidia HWinf-CS Sites:	---	OCF Sites:	---
Recursion Pharma Sites:	---	SFW Sites:	---
SNIC sites:	---	Tzag Elita Sites:	---
Linux Distro:	---	Machine Name:
CLE Version:		Version Fixed:
Target Release:	---	DevPrio:	---
Emory-Cloud Sites:	---

Description paull 2016-08-12 07:54:51 MDT

Hi,

I would like to know if there was an option to limit the amount of nodes for a job array used at one time?

Ex: Job 1111_[1-100], max nodes used at 1 time = 10. Therefore, at max, while this array is in queue, only 10/100 jobs are running.

Submit option or update to job after submission? I would like the user to be able to control this.

Thanks,

Paul

Comment 3 Moe Jette 2016-08-12 08:24:58 MDT

Would a mechanism to limit the number of tasks from a job array that can be running at any time be sufficient? Relevant documentation below:

"A maximum number of simultaneously running tasks from the job array may be specified using a "%" separator. For example "--array=0-15%4" will limit the number of simultaneously running tasks from this job array to 4."

This count can be altered on job already submitted using the scontrol command. for example: scontrol update jobid=1234 ArrayTaskThrottle=4
ArrayTaskThrottle=<count>
    Speciify the maximum number of tasks in a job array that can execute at the same time. Set the count to zero in order to eliminate any limit. The task throttle count for a job array is reported as part of its ArrayTaskId field, preceded with a percent sign. For example "ArrayTaskId=1-10%2" indicates the maximum number of running tasks is limited to 2.

Comment 5 paull 2016-08-12 10:40:48 MDT

Hi Moe,

Thanks for your response. This may be exactly what the doctor ordered, but I will follow up after getting the green light from our team. 

Thanks,

Paul

Also, would it be possible to get some insight on ticket #2965?

Comment 6 Moe Jette 2016-08-15 08:23:05 MDT

(In reply to paull from comment #5)
> Also, would it be possible to get some insight on ticket #2965?

The last comment there was from Tim asking for some logs...

Comment 7 paull 2016-08-15 10:03:39 MDT

Moe,

Thanks for your help. I don't have an update as of yet on if this is what they team needs or not. As soon as I do I will update this ticket.

Thanks!

Paul

Comment 8 paull 2016-08-25 11:16:05 MDT

Moe,

Since we will be updating soon, I don't see a need for this to stay open. The solution you provided has been helpful.

Thanks!

Paul