Ticket 15865 - Job Prioritization Suggestions
Summary: Job Prioritization Suggestions
Status: RESOLVED INFOGIVEN
Alias: None
Product: Slurm
Classification: Unclassified
Component: Scheduling (show other tickets)
Version: 22.05.2
Hardware: Linux Linux
: 4 - Minor Issue
Assignee: Carlos Tripiana Montes
QA Contact:
URL:
Depends on:
Blocks:
 
Reported: 2023-01-25 08:41 MST by rvasquez2
Modified: 2023-02-20 10:53 MST (History)
1 user (show)

See Also:
Site: Dow Chemical
Slinky Site: ---
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
Google sites: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
NoveTech Sites: ---
Nvidia HWinf-CS Sites: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Tzag Elita Sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed:
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this ticket.
Description rvasquez2 2023-01-25 08:41:45 MST 
Is there a recommendation on how to prioritize jobs to minimize queued time+time limit?

For example, if 2 jobs are queued at the same time for the same amount of resources, the one with the shorter time limit will run first.

The tiering of priority based on partition feels too severe
Comment 1 Carlos Tripiana Montes 2023-01-25 09:02:11 MST 
Hi,

Have you checked [1] plus [2]?

So, in slurm.conf:

PriorityType=priority/multifactor
PriorityFavorSmall=YES
PriorityFlags=SMALL_RELATIVE_TO_TIME

This is as close as what you've requested as I am able to think of right now, to be fair. But I think it would work like a charm.

Remember to restart slurmctld.

Cheers,
Carlos.

[1] https://slurm.schedmd.com/slurm.conf.html#OPT_PriorityFavorSmall
[2] https://slurm.schedmd.com/slurm.conf.html#OPT_SMALL_RELATIVE_TO_TIME
Comment 2 Mike 2023-01-25 09:45:28 MST 
Hi Carlos,

Is there a time component in PriorityFavorSmall as well?  The documentation doesn't seem to indicate one so we were previously using NO to fill resources with the largest jobs possible and then backfill with the small jobs where free resources exist.  

Thanks,
Mike
Comment 3 Carlos Tripiana Montes 2023-01-26 01:51:58 MST 
Hi Mike,

Revisiting Comment 0 with what you added in Comment 2 gives another picture different from what I was thinking you were looking for.

All in all, if I understood it right:

1. Favour big jobs over small ones.
2. Favour shorter jobs over longer ones.

Which feels like the example given in [1]:

"The PriorityFlags value of SMALL_RELATIVE_TO_TIME alters this behavior as follows. The job size in CPUs is divided by the time limit in minutes. The result is divided by the total number of CPUs in the system. Thus a full-system job with a time limit of one will receive a job size factor of 1.0, while a tiny job with a large time limit will receive a job size factor close to 0.0."

To accomplish this, slurm.conf must have:

PriorityType=priority/multifactor
PriorityFavorSmall=NO
PriorityFlags=SMALL_RELATIVE_TO_TIME

You see it's the same as in Comment 1, but changing PriorityFavorSmall from YES back to NO (you can also remove the line, because NO is the default value).

Hopefully, now I've got the point and my suggestions will be helpful.

Cheers,
Carlos.

[1] https://slurm.schedmd.com/priority_multifactor.html#jobsize
Comment 4 Carlos Tripiana Montes 2023-02-14 06:20:14 MST 
Hey Mike,

Do you need anything else from our side here? Otherwise, you might consider to close the issue as resolved/infovigen.

Thanks,
Carlos.
Comment 5 Mike 2023-02-20 10:53:04 MST 
Carlos, I think we have all the info we need.  Thanks!