Ticket 14073 - Possibility of resuming nodes not needed by job
Summary: Possibility of resuming nodes not needed by job
Status: RESOLVED FIXED
Alias: None
Product: Slurm
Classification: Unclassified
Component: Other (show other tickets)
Version: 22.05.x
Hardware: Linux Linux
: 4 - Minor Issue
Assignee: Scott Hilton
QA Contact: Brian Christiansen
URL:
: 14138 (view as ticket list)
Depends on:
Blocks:
 
Reported: 2022-05-16 10:27 MDT by Scott Hilton
Modified: 2022-05-24 13:57 MDT (History)
1 user (show)

See Also:
Site: SchedMD
Slinky Site: ---
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
Google sites: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
NoveTech Sites: ---
Nvidia HWinf-CS Sites: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Tzag Elita Sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed: 21.08.9 22.05.0rc2 23.02.0pre1
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---


Attachments
v1 (733 bytes, patch)
2022-05-16 10:28 MDT, Scott Hilton
Details | Diff

Note You need to log in before you can comment on or make changes to this ticket.
Description Scott Hilton 2022-05-16 10:27:13 MDT
In _do_power_work() in power_save.c

i_first = bit_ffs(need_resume_bitmap);
if (i_first >= 0)
	i_last = bit_fls(need_resume_bitmap);
else
	i_last = i_first - 1;
for (i = i_first; i <= i_last; i++) {
	if ((resume_rate == 0) || (resume_cnt < resume_rate)) {
		resume_cnt++;
		resume_cnt_f++;

		bit_set(job_power_node_bitmap, i);
		bit_set(to_resume_bitmap, i);
	}
}

This for loop obviously needs this check:		
if (!bit_test(need_resume_bitmap, i))
	continue;

Or else any non continuous set of nodes requested to resume could get nodes in between them resuming instead of the requested nodes.
Comment 1 Scott Hilton 2022-05-16 10:28:59 MDT
Created attachment 25040 [details]
v1
Comment 2 Scott Hilton 2022-05-16 11:03:52 MDT
Reproducer:

slurm.conf
ResumeRate=2

srun --nodelist=node0,node3 hostname

sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
debug        up   infinite      1   mix# node0
debug        up   infinite      1  idle# node1
debug        up   infinite      1   mix~ node3
Comment 3 Brian Christiansen 2022-05-17 08:52:04 MDT
Thanks Scott! Great find!

fd1ea63a6d Fix resuming nodes not part of an allocation
Comment 4 Skyler Malinowski 2022-05-24 13:57:01 MDT
*** Ticket 14138 has been marked as a duplicate of this ticket. ***