Ticket 14138

Summary:	Power saving resuming drained nodes, changes reason
Product:	Slurm	Reporter:	Dylan Simon <dsimon>
Component:	Other	Assignee:	Skyler Malinowski <smalinowski>
Status:	RESOLVED DUPLICATE	QA Contact:
Severity:	4 - Minor Issue
Priority:	---
Version:	21.08.8
Hardware:	Linux
OS:	Linux
Site:	Simons Foundation & Flatiron Institute	Slinky Site:	---
Alineos Sites:	---	Atos/Eviden Sites:	---
Confidential Site:	---	Coreweave sites:	---
Cray Sites:	---	DS9 clusters:	---
Google sites:	---	HPCnow Sites:	---
HPE Sites:	---	IBM Sites:	---
NOAA SIte:	---	NoveTech Sites:	---
Nvidia HWinf-CS Sites:	---	OCF Sites:	---
Recursion Pharma Sites:	---	SFW Sites:	---
SNIC sites:	---	Tzag Elita Sites:	---
Linux Distro:	---	Machine Name:
CLE Version:		Version Fixed:
Target Release:	---	DevPrio:	---
Emory-Cloud Sites:	---

Description Dylan Simon 2022-05-23 09:07:42 MDT

We have power saving enabled:

SuspendTime=900
SuspendTimeout=60
ResumeTimeout=300
SuspendProgram=/mnt/sw/fi/bin/slurm-suspend
ResumeProgram=/mnt/sw/fi/bin/slurm-resume

Since upgrading from 21.08.6 to 21.08.8, we've noticed that slurm sometimes runs the ResumeProgram on a big range of nodes:

Mon May 23 10:33:19 EDT 2022 /mnt/sw/fi/bin/slurm-resume worker[3000-3155]

This sometimes includes nodes that had been marked as DRAIN or DOWN or (DRAIN+DOWN):

[2022-05-22T09:10:43.972] update_node: node worker3133 reason set to: #868 power
[2022-05-22T09:10:43.972] update_node: node worker3133 state set to DOWN

(This actually isn't new behavior, so far.) Some of these nodes are basically dead and won't power on, but then after the ResumeTimeout:

[2022-05-23T10:38:20.843] node worker3133 not resumed by ResumeTimeout(300) - marking down and power_save

It changes the Reason on the node to "ResumeTimeout reached" and loses the original reason.  Is there a way to prevent this?  I notice there's a new ResumeFailProgram setting we're not using -- can it be used somehow?  Or is there a way to not resume/suspend DRAINed nodes altogether with SuspendExc something, or some other state we could set these nodes to to avoid this?  Thanks!

Comment 1 Skyler Malinowski 2022-05-24 10:21:41 MDT

> Since upgrading from 21.08.6 to 21.08.8, we've noticed that slurm sometimes
> runs the ResumeProgram on a big range of nodes:
> 
> Mon May 23 10:33:19 EDT 2022 /mnt/sw/fi/bin/slurm-resume worker[3000-3155]

In 22.05.0rc2, commit fd1ea63a6d4 fixes a regression where additional nodes were being powered up by power save. Can you confirm you are affected by this issue by submitting a job requiring two nodes that flank another powered down node.
> sbatch -w node[0,2] --wrap=hostname


> I notice there's a new ResumeFailProgram setting we're not using -- can
> it be used somehow?

I am not sure what you are referring to.


> It changes the Reason on the node to "ResumeTimeout reached" and loses the
> original reason.  Is there a way to prevent this?

No. The node reason is for recording the last admin or system message. If you would like a persistent admin message, consider using comment instead.
> scontrol update nodename=node0 comment=test

These messages will persist until an admin changes it.


> Or is there a way to not resume/suspend DRAINed nodes altogether with
> SuspendExc something, or some other state we could set these nodes to
> avoid this?

SuspendExc will certainly prevent those nodes from being automatically managed by power_save, hence they will not be powered down when idle for too long. As far as I can tell, SuspendExc is the only way to guarantee the node will not be automatically managed by power_save.

Comment 2 Dylan Simon 2022-05-24 13:45:18 MDT

Ah, yes, I submitted a job "-w worker1109,worker1111" while worker1110 was DOWN, and it ran "/mnt/sw/fi/bin/slurm-resume worker[1109-1111]" and then eventually timed out, so it looks like bug #14073 does trigger this issue.  It looks like that fix is already on slurm 21.08 branch as well.  We'll try applying it and hopefully will avoid this altogether. Feel free to close and we can re-open if not.  Thanks!

Comment 3 Skyler Malinowski 2022-05-24 13:57:01 MDT

Upgrade to 21.08.9 (when available) or another version with the patch. If you still see issues. then we can certainly address them.

Best,
Skyler

*** This ticket has been marked as a duplicate of ticket 14073 ***