Ticket 15865

Summary:	Job Prioritization Suggestions
Product:	Slurm	Reporter:	rvasquez2
Component:	Scheduling	Assignee:	Carlos Tripiana Montes <tripiana>
Status:	RESOLVED INFOGIVEN	QA Contact:
Severity:	4 - Minor Issue
Priority:	---	CC:	mfazio1
Version:	22.05.2
Hardware:	Linux
OS:	Linux
Site:	Dow Chemical	Slinky Site:	---
Alineos Sites:	---	Atos/Eviden Sites:	---
Confidential Site:	---	Coreweave sites:	---
Cray Sites:	---	DS9 clusters:	---
Google sites:	---	HPCnow Sites:	---
HPE Sites:	---	IBM Sites:	---
NOAA SIte:	---	NoveTech Sites:	---
Nvidia HWinf-CS Sites:	---	OCF Sites:	---
Recursion Pharma Sites:	---	SFW Sites:	---
SNIC sites:	---	Tzag Elita Sites:	---
Linux Distro:	---	Machine Name:
CLE Version:		Version Fixed:
Target Release:	---	DevPrio:	---
Emory-Cloud Sites:	---

Description rvasquez2 2023-01-25 08:41:45 MST

Is there a recommendation on how to prioritize jobs to minimize queued time+time limit?

For example, if 2 jobs are queued at the same time for the same amount of resources, the one with the shorter time limit will run first.

The tiering of priority based on partition feels too severe

Comment 1 Carlos Tripiana Montes 2023-01-25 09:02:11 MST

Hi,

Have you checked [1] plus [2]?

So, in slurm.conf:

PriorityType=priority/multifactor
PriorityFavorSmall=YES
PriorityFlags=SMALL_RELATIVE_TO_TIME

This is as close as what you've requested as I am able to think of right now, to be fair. But I think it would work like a charm.

Remember to restart slurmctld.

Cheers,
Carlos.

[1] https://slurm.schedmd.com/slurm.conf.html#OPT_PriorityFavorSmall
[2] https://slurm.schedmd.com/slurm.conf.html#OPT_SMALL_RELATIVE_TO_TIME

Comment 2 Mike 2023-01-25 09:45:28 MST

Hi Carlos,

Is there a time component in PriorityFavorSmall as well?  The documentation doesn't seem to indicate one so we were previously using NO to fill resources with the largest jobs possible and then backfill with the small jobs where free resources exist.  

Thanks,
Mike

Comment 3 Carlos Tripiana Montes 2023-01-26 01:51:58 MST

Hi Mike,

Revisiting Comment 0 with what you added in Comment 2 gives another picture different from what I was thinking you were looking for.

All in all, if I understood it right:

1. Favour big jobs over small ones.
2. Favour shorter jobs over longer ones.

Which feels like the example given in [1]:

"The PriorityFlags value of SMALL_RELATIVE_TO_TIME alters this behavior as follows. The job size in CPUs is divided by the time limit in minutes. The result is divided by the total number of CPUs in the system. Thus a full-system job with a time limit of one will receive a job size factor of 1.0, while a tiny job with a large time limit will receive a job size factor close to 0.0."

To accomplish this, slurm.conf must have:

PriorityType=priority/multifactor
PriorityFavorSmall=NO
PriorityFlags=SMALL_RELATIVE_TO_TIME

You see it's the same as in Comment 1, but changing PriorityFavorSmall from YES back to NO (you can also remove the line, because NO is the default value).

Hopefully, now I've got the point and my suggestions will be helpful.

Cheers,
Carlos.

[1] https://slurm.schedmd.com/priority_multifactor.html#jobsize

Comment 4 Carlos Tripiana Montes 2023-02-14 06:20:14 MST

Hey Mike,

Do you need anything else from our side here? Otherwise, you might consider to close the issue as resolved/infovigen.

Thanks,
Carlos.

Comment 5 Mike 2023-02-20 10:53:04 MST

Carlos, I think we have all the info we need.  Thanks!