Ticket 14081

Summary: preemption question
Product: Slurm Reporter: Jeff Haferman <jlhaferm>
Component: SchedulingAssignee: Director of Support <support>
Status: RESOLVED INFOGIVEN QA Contact:
Severity: 4 - Minor Issue    
Priority: ---    
Version: 20.02.5   
Hardware: Linux   
OS: Linux   
Site: NPS HPC Slinky Site: ---
Alineos Sites: --- Atos/Eviden Sites: ---
Confidential Site: --- Coreweave sites: ---
Cray Sites: --- DS9 clusters: ---
Google sites: --- HPCnow Sites: ---
HPE Sites: --- IBM Sites: ---
NOAA SIte: --- NoveTech Sites: ---
Nvidia HWinf-CS Sites: --- OCF Sites: ---
Recursion Pharma Sites: --- SFW Sites: ---
SNIC sites: --- Tzag Elita Sites: ---
Linux Distro: --- Machine Name:
CLE Version: Version Fixed:
Target Release: --- DevPrio: ---
Emory-Cloud Sites: ---

Description Jeff Haferman 2022-05-16 18:59:07 MDT 
This is my first time working with preemption. I'm following along at:
https://slurm.schedmd.com/preempt.html

I'm looking at the example there:

[user@n16 ~]$ sinfo
PARTITION AVAIL  TIMELIMIT NODES  STATE NODELIST
active*      up   infinite     5   idle n[12-16]
hipri        up   infinite     5   idle n[12-16]

[user@n16 ~]$ grep PartitionName /shared/slurm/slurm.conf
PartitionName=DEFAULT OverSubscribe=FORCE:1 Nodes=n[12-16]
PartitionName=active PriorityTier=1 Default=YES
PartitionName=hipri  PriorityTier=2


Let's say I want n[12-16] to continue to be the DEFAULT partition, but I only want n15 and n16 to be in the "hipri" partition, and I only want jobs submitted to the DEFAULT partition to land on n15 or n16 IF AND ONLY IF no other nodes in the DEFAULT partition are available.

What changes would need to be made to the slurm.conf for this to happen?
Comment 1 Caden Ellis 2022-05-17 16:08:20 MDT 
Based on what what you have described here, it sounds like you want to use something called the "Condo Model." 

Bug#1032Comment#1 
This example uses Node Weights, Partition Priorities, and an ACL on higher priority partitions.
In your case, you could specify the whole node range in the default queue, and have the 2 other nodes in the high priority queue.

Here is a very basic example of what I think you are wanting:

PreemptMode=requeue
PreemptType=preempt/partition_prio

PartitionName=hirpio Nodes=tux[15-16] PriorityTier=2 AllowGroups=PI_B
PartitionName=DEFAULT  Nodes=tux[12-16] PriorityTier=1 Default=YES 
              AllowGroups=PI_A,PI_B

The above snippet will allow the use of all of the nodes 12-16, while filling 12-14 first, but only if we specify node weights (not shown above). 

https://slurm.schedmd.com/slurm.conf.html#OPT_Weight

The node weights determine which nodes to give precedence to for job placement, the priority tiers are for job preemption. In the above example, Group PI_B can preempt jobs running in the DEFAULT queue. It sounds like node weights have more to do with what you are asking than preemption.

Between that linked bug and the node_weights doc link, I believe you should have what you need.
Comment 2 Jeff Haferman 2022-05-17 17:06:18 MDT 
Yes, that's exactly what I want... "condo model". Thank you so much for the reply, I will work on implementing it!
Comment 3 Caden Ellis 2022-05-18 11:26:31 MDT 
If you have any follow-up questions, do not hesitate to log these in a new bug.