Ticket 3078

I've confirmed this a few times today. Twice when I use `service slurm restart` on the controller, the excluded node stays excluded from suspension despite being idle for much longer than SuspendTime. Twice when I used `service slurm reload` it is immediately suspended.

This was from the restart:

[2016-09-13T18:37:13.346] Terminate signal (SIGINT or SIGTERM) received
[2016-09-13T18:37:13.346] debug:  sched: slurmctld terminating
[2016-09-13T18:37:13.346] debug3: _slurmctld_rpc_mgr shutting down
[2016-09-13T18:37:13.384] Saving all slurm state
[2016-09-13T18:37:13.386] debug3: Writing job id 21 to header record of job_state file
[2016-09-13T18:37:13.392] debug3: _slurmctld_background shutting down
[2016-09-13T18:37:13.395] debug:  mcs none plugin fini
[2016-09-13T18:37:14.099] debug3: layouts: layouts_fini()...
[2016-09-13T18:37:14.099] debug:  layouts/base: dumping 6 records into state file
[2016-09-13T18:37:14.100] layouts: all layouts are now unloaded.
[2016-09-13T18:37:14.100] Unable to remove pidfile '/var/run/slurmctld.pid': Permission denied
[2016-09-13T18:37:14.435] Job accounting information stored, but details not gathered
[2016-09-13T18:37:14.436] slurmctld version 16.05.4 started on cluster aurecc
[2016-09-13T18:37:14.439] layouts: no layout to initialize
[2016-09-13T18:37:14.445] layouts: loading entities/relations information
[2016-09-13T18:37:14.445] Recovered state of 3 nodes
[2016-09-13T18:37:14.445] Recovered information about 0 jobs
[2016-09-13T18:37:14.445] Recovered state of 0 reservations
[2016-09-13T18:37:14.445] read_slurm_conf: backup_controller not specified.
[2016-09-13T18:37:14.446] Running as primary controller
[2016-09-13T18:37:14.446] No parameter for mcs plugin, default values set
[2016-09-13T18:37:14.446] mcs: MCSParameters = (null). ondemand set.
[2016-09-13T18:37:15.449] power_save module, excluded nodes slurm0
..
[2016-09-13T18:37:14.445] Recovered state of 3 nodes
[2016-09-13T18:37:14.445] Recovered information about 0 jobs
[2016-09-13T18:37:14.445] Recovered state of 0 reservations
[2016-09-13T18:37:14.445] read_slurm_conf: backup_controller not specified.
[2016-09-13T18:37:14.446] Running as primary controller
[2016-09-13T18:37:14.446] No parameter for mcs plugin, default values set
[2016-09-13T18:37:14.446] mcs: MCSParameters = (null). ondemand set.
[2016-09-13T18:37:15.449] power_save module, excluded nodes slurm0
[2016-09-13T18:37:15.449] Power save mode: 2 nodes

I only have three nodes defined, slurm0 being excluded as above.

This is from the reload (from 21:37):

[2016-09-13T21:28:38.147] Power save mode: 2 nodes
[2016-09-13T21:37:27.803] Reconfigure signal (SIGHUP) received
[2016-09-13T21:37:27.807] restoring original state of nodes
[2016-09-13T21:37:27.807] restoring original partition state
[2016-09-13T21:37:27.808] read_slurm_conf: backup_controller not specified.
[2016-09-13T21:37:27.808] No parameter for mcs plugin, default values set
[2016-09-13T21:37:27.808] mcs: MCSParameters = (null). ondemand set.
[2016-09-13T21:37:28.683] SchedulerParameters=default_queue_depth=100,max_rpc_cnt=0,max_sched_time=2,partition_job_depth=0,sched_max_job_start=0,sched_min_interval=0
[2016-09-13T21:39:28.232] Power save mode: 3 nodes

and my SuspendProgram script notes:

09/13/16T21:37:28 (suspend.sh) - SuspendProgram /usr/local/slurm/scripts/suspend.sh invoked with args slurm0

Regards
Jeff

Hi Jacob

It is not on any specific site, and certainly not on one that is supported by schedmd. I just thought I'd mention it. I have been setting this up on an instance of the IBM Softlayer cloud, sometimes here in Melbourne, other times in Toronto. This being a prototyping exercise, the host doesn't stay up very long, either.

Regards
Jeff

Also seen with *scontrol reconfigure*:

Started the service last night:

[2016-09-15T23:15:30.053] power_save module, excluded nodes slurm0
[2016-09-15T23:15:30.053] Power save mode: 10 nodes
..

Launched a job, took a while but the instance did come up:

[2016-09-15T23:18:24.074] backfill: Started JobId=2 in main on slurm0
..
[2016-09-15T23:50:45.256] Node slurm0 now responding
[2016-09-15T23:50:59.868] job_time_limit: Configuration for job 2 is complete
..
[2016-09-15T23:52:12.962] sched: Allocate JobID=4 NodeList=slurm0 #CPUs=1 Partition=main
[2016-09-15T23:52:16.685] _pick_step_nodes: Configuration for job 4 is complete
[2016-09-15T23:53:01.957] job_complete: JobID=4 State=0x1 NodeCnt=1 WEXITSTATUS 0
[2016-09-15T23:53:01.959] job_complete: JobID=4 State=0x8003 NodeCnt=1 done
[2016-09-15T23:57:59.464] Power save mode: 9 nodes
..

Ran some jobs on it and then left it alone. The excluded node stayed up all night.

Then I updated slurm.conf this morning:

[2016-09-16T12:02:16.747] error: Node slurm0 appears to have a different slurm.conf than the slurmctld.  This could cause issues with communication and functionality.  Please review both files and make sure they are the same.  If this is expected ignore, and set DebugFlags=NO_CONF_HASH in your slurm.conf.

so reconfigured. 

[2016-09-16T12:02:27.463] Processing RPC: REQUEST_RECONFIGURE from uid=0

SuspendTime (20) minutes later, a request came in to suspend the excluded node:

2016-09-16 12:23:05,886 [INFO] Suspend operation for node slurm0

Oddly enough, no slurmctld logs at around this time, until I restarted slurmctld:

[2016-09-16T12:22:57.761] Power save mode: 9 nodes
[2016-09-16T12:33:26.578] Terminate signal (SIGINT or SIGTERM) received

Hope this helps.

- Jeff

I'm seeing the exact same behaviour with 16.05.3 testing elastic computing. 

After a scontrol reconfigure or a service reload, slurmctld seems to forget about the SuspendExcNodes and starts to (try to) suspend them, thus putting the excluded nodes into a power saving state.

Jeff,

You might want to retest this with Slurm 16.05.5, we've rolled it out on the machine I'm helping Linh with and it's looking good so far (touch wood).

Can't see a commit that would be relevant though!

Hiya Chris, didn't help. I upgraded to 16.05.5, started things up, ran a job that woke up the excluded node:

[2016-10-17T20:03:47.127] slurmctld version 16.05.5 started on cluster melslurm
..
[2016-10-17T20:03:48.794] power_save module, excluded nodes melslurm1
[2016-10-17T20:05:34.531] Power save mode: 3 nodes
..
[2016-10-17T23:43:07.790] _slurm_rpc_submit_batch_job JobId=2 usec=786
..
[2016-10-17T23:56:09.121] Power save mode: 2 nodes
..
[2016-10-19T11:08:31.506] Power save mode: 2 nodes

Then hours later I reconfigured, 

[2016-10-19T11:13:38.239] Processing RPC: REQUEST_RECONFIGURE from uid=0
[2016-10-19T11:13:38.240] restoring original state of nodes
[2016-10-19T11:13:38.240] _preserve_plugins: backup_controller not specified
[2016-10-19T11:13:38.240] No parameter for mcs plugin, default values set
[2016-10-19T11:13:38.240] mcs: MCSParameters = (null). ondemand set.
[2016-10-19T11:13:38.241] _slurm_rpc_reconfigure_controller: completed usec=2087
..

and slurmctld immediately called for a suspend operation on the excluded node. This was via the SuspendProgram log file:

2016-10-19T11:13:38 suspend.sh - SuspendProgram /usr/local/slurm/scripts/suspend.sh invoked [args melslurm1] ()

So the bug is still there: when slurmctld completes a reconfigure operation, it will have forgotten about SuspendExcNodes. Only a proper service restart brings it back.

On 19/10/16 11:25, bugs@schedmd.com wrote:

> So the bug is still there: when slurmctld completes a reconfigure operation, it
> will have forgotten about SuspendExcNodes. Only a proper service restart brings
> it back.

:-(

So I guess in lieu of either IBM or Research Platforms taking out a
support contract it'll come down to either you or them digging through
the source code and coming up with a patch for SchedMD!

cheers,
Chris

Or we block the suspension of the excluded node -- in effect, duplicating the feature externally. I wonder if other parts of the power_save module end up being discarded though..

On 19/10/16 13:42, bugs@schedmd.com wrote:

> Comment # 8 on bug 3078 from Jeff Tan
> 
> Or we block the suspension of the excluded node -- in effect, duplicating the
> feature externally. I wonder if other parts of the power_save module end up
> being discarded though..

I think Linh did that for Research Platforms which at least stops
OpenStack from deleting the node, but IIRC Slurm will never use it again
as it is confused about its state.

cheers,
Chris

> Slurm will never use it again
> as it is confused about its state.

Sadly, that seems to be the case. I thought there was a way to get it back in without actually powering it down then up again, all at the instigation of slurmctld, but no such workaround I can see.

(In reply to Jeff Tan from comment #10)

> Sadly, that seems to be the case. I thought there was a way to get it back
> in without actually powering it down then up again, all at the instigation
> of slurmctld, but no such workaround I can see.

One thing we tried last week was to set the node state to either POWER_DOWN or POWER_UP (depending on state) to tell Slurm to change its state.  Worth a shot.

We've had to deconfigure the suspend/resume and cloud capabilities in Slurm to try and stabilise the cluster, hopefully at some point this code will work again. :-(

I can reproduce this, SuspendExcNodes is ignored on SIGHUP.

Chris, Jefff - following commit addressed for 16.05.7 fixes this:

https://github.com/SchedMD/slurm/commit/bb06dd65c

Closing the bug. Please, reopen if you encounter further issues.

Thanks! It works! :-)