Ticket 15185

Summary:	Dynamically update SuspendExcNodes and SuspendExcParts during runtime
Product:	Slurm	Reporter:	Chrysovalantis Paschoulas <c.paschoulas>
Component:	Other	Assignee:	Scott Hilton <scott>
Status:	RESOLVED FIXED	QA Contact:
Severity:	5 - Enhancement
Priority:	---	CC:	bas.vandervlies, brian, skyler
Version:	21.08.8
Hardware:	Linux
OS:	Linux
Site:	Jülich	Slinky Site:	---
Alineos Sites:	---	Atos/Eviden Sites:	---
Confidential Site:	---	Coreweave sites:	---
Cray Sites:	---	DS9 clusters:	---
Google sites:	---	HPCnow Sites:	---
HPE Sites:	---	IBM Sites:	---
NOAA SIte:	---	NoveTech Sites:	---
Nvidia HWinf-CS Sites:	---	OCF Sites:	---
Recursion Pharma Sites:	---	SFW Sites:	---
SNIC sites:	---	Tzag Elita Sites:	---
Linux Distro:	---	Machine Name:
CLE Version:		Version Fixed:	23.02.0rc1
Target Release:	23.11	DevPrio:	1 - Paid
Emory-Cloud Sites:	---

Description Chrysovalantis Paschoulas 2022-10-16 09:23:19 MDT

I have already opened Bug 15184 where I presented our hard requirement to be able to skip powering down nodes that are drained, and here I would like to talk about some other ideas I have to improve the whole power saving mechanism.

I think it would be really nice if it was possible to update during runtime SuspendExcNodes and SuspendExcParts and make them restart-proof.

The interface could be like:
```
scontrol suspendexcnodes=+/-<nodelist>
scontrol suspendexcparts=+/-<partitions>
```

What is in SuspendExcNodes and SuspendExcParts should be static and unchangeable, but we could add nodes and partitions dynamically (and be able to revome them later). Also these data should be stored in state files in order to be restorable after restarting slurmctld.

What do you think?

Comment 1 Bas van der Vlies 2022-10-18 05:17:53 MDT

This is really a nice idea  dynamically update the exclude config for power saving mechanism.

Comment 6 Scott Hilton 2023-01-16 16:05:13 MST

SuspendExcNodes has a peculiar kind of nodelist with the optional ":" separator. This is used to specify groups of nodes from which a certain number should stay online.
See: https://slurm.schedmd.com/slurm.conf.html#OPT_SuspendExcNodes

Is it important to you to be able to add and remove from lists with this special ":" syntax? If so what is specifically needed for your workflow?

-Scott

Comment 7 Chrysovalantis Paschoulas 2023-01-17 01:25:32 MST

(In reply to Scott Hilton from comment #6)
> SuspendExcNodes has a peculiar kind of nodelist with the optional ":"
> separator. This is used to specify groups of nodes from which a certain
> number should stay online.
> See: https://slurm.schedmd.com/slurm.conf.html#OPT_SuspendExcNodes
> 
> Is it important to you to be able to add and remove from lists with this
> special ":" syntax? If so what is specifically needed for your workflow?
> 
> -Scott

Hi Scott!

No, for our case I would say that this feature of ":" is not needed. As far as I can imagine we will need to exclude from suspension only specific nodes, e.g. because we will want to use them for various reasons (like reserving them for a course or doing some tests on them, running the testsuite etc..) or keeping them online for doing some maintenance, HW work, etc..

Cheers,
Valantis

Comment 13 Scott Hilton 2023-02-01 14:59:07 MST

Valantis,

We have completed this feature and it should be part of release 23.02. See commits fc5ec8c83f - 77c1c7d7ae.


-Scott

Comment 14 Chrysovalantis Paschoulas 2023-02-02 02:08:16 MST

(In reply to Scott Hilton from comment #13)
> Valantis,
> 
> We have completed this feature and it should be part of release 23.02. See
> commits fc5ec8c83f - 77c1c7d7ae.
> 
> 
> -Scott

Hi Scott, that's great!

I see that we will be able to dynamically update the excluded states too :)

Thank you very much!

-Valantis