Ticket 14073

Summary: Possibility of resuming nodes not needed by job
Product: Slurm Reporter: Scott Hilton <shilton>
Component: OtherAssignee: Scott Hilton <shilton>
Status: RESOLVED FIXED QA Contact: Brian Christiansen <bchristianse>
Severity: 4 - Minor Issue    
Priority: --- CC: dsimon
Version: 22.05.x   
Hardware: Linux   
OS: Linux   
Site: SchedMD Slinky Site: ---
Alineos Sites: --- Atos/Eviden Sites: ---
Confidential Site: --- Coreweave sites: ---
Cray Sites: --- DS9 clusters: ---
Google sites: --- HPCnow Sites: ---
HPE Sites: --- IBM Sites: ---
NOAA SIte: --- NoveTech Sites: ---
Nvidia HWinf-CS Sites: --- OCF Sites: ---
Recursion Pharma Sites: --- SFW Sites: ---
SNIC sites: --- Tzag Elita Sites: ---
Linux Distro: --- Machine Name:
CLE Version: Version Fixed: 21.08.9 22.05.0rc2 23.02.0pre1
Target Release: --- DevPrio: ---
Emory-Cloud Sites: ---
Attachments: v1

Description Scott Hilton 2022-05-16 10:27:13 MDT 
In _do_power_work() in power_save.c

i_first = bit_ffs(need_resume_bitmap);
if (i_first >= 0)
	i_last = bit_fls(need_resume_bitmap);
else
	i_last = i_first - 1;
for (i = i_first; i <= i_last; i++) {
	if ((resume_rate == 0) || (resume_cnt < resume_rate)) {
		resume_cnt++;
		resume_cnt_f++;

		bit_set(job_power_node_bitmap, i);
		bit_set(to_resume_bitmap, i);
	}
}

This for loop obviously needs this check:		
if (!bit_test(need_resume_bitmap, i))
	continue;

Or else any non continuous set of nodes requested to resume could get nodes in between them resuming instead of the requested nodes.
Comment 1 Scott Hilton 2022-05-16 10:28:59 MDT 
Created attachment 25040 [details]
v1
Comment 2 Scott Hilton 2022-05-16 11:03:52 MDT 
Reproducer:

slurm.conf
ResumeRate=2

srun --nodelist=node0,node3 hostname

sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
debug        up   infinite      1   mix# node0
debug        up   infinite      1  idle# node1
debug        up   infinite      1   mix~ node3
Comment 3 Brian Christiansen 2022-05-17 08:52:04 MDT 
Thanks Scott! Great find!

fd1ea63a6d Fix resuming nodes not part of an allocation
Comment 4 Skyler Malinowski 2022-05-24 13:57:01 MDT 
*** Ticket 14138 has been marked as a duplicate of this ticket. ***