Ticket 14721

Is this happening on a machine where you can share slurmd/slurmctld logs?

(In reply to Nate Rini from comment #1)
> Is this happening on a machine where you can share slurmd/slurmctld logs?

Nate , we have seen a couple  instances that we are trying to chase down the cause for our Shasta system in the Open , so we might be able to share some of the info off that.. currently i don't have one to send you yet though .. I am in a holding pattern waiting for it to happen again on chicoma , if its going to ..

question will setting up the nodehealthscript script work like this.. I do have a patch to increase its time out value ..

we had another incident on our secure XC machine , but it seems linked to  power issue that ended up requiring a full reboot of the cluster , so i am writing that one off.

i am trying to be proactive and see if i can move the administrative things not related directly to the job out of the prolog and epilog

(In reply to mike coyne from comment #2)
> question will setting up the nodehealthscript script work like this.. I do
> have a patch to increase its time out value ..
Please attach the patch so I can test with it.
 
> i am trying to be proactive and see if i can move the administrative things
> not related directly to the job out of the prolog and epilog
understood

(In reply to mike coyne from comment #0)
> will this execute it between one jobs epilog and the next ones prolog once
> per hour , and if i have a job that say lasts for 16 hours , will it run it
> after the completion of the long 16 hour job  but before the next one.
The health_check is called by the cronlike _slurmctld_background() thread in slurmctld. It then filters the node list from the nodes that are not DOWN. There is no logic currently to ensure that it is executed between jobs. It may execute between jobs but there is nothing making sure that it will happen. Depending on how busy slurmctld is at the time, it is even less likely to execute between jobs.

Created attachment 26263 [details]
node health check time out patch

this is from 21.8.6+ i have not ported it to 22.5 yet .

(In reply to Nate Rini from comment #4)
> (In reply to mike coyne from comment #0)
> > will this execute it between one jobs epilog and the next ones prolog once
> > per hour , and if i have a job that say lasts for 16 hours , will it run it
> > after the completion of the long 16 hour job  but before the next one.
> The health_check is called by the cronlike _slurmctld_background() thread in
> slurmctld. It then filters the node list from the nodes that are not DOWN.
> There is no logic currently to ensure that it is executed between jobs. It
> may execute between jobs but there is nothing making sure that it will
> happen. Depending on how busy slurmctld is at the time, it is even less
> likely to execute between jobs.

Please note that there is currently bug#12004 open to improving the functionality of health_check. I suggest taking a look.

If the goal is to ensure that the health_check runs after every job, then calling the health_check script via the epilog is the suggested route. It is already required that the health_check script request that the node gets closed so it should run there without any issues.

(In reply to Nate Rini from comment #10)
> (In reply to Nate Rini from comment #4)
> > (In reply to mike coyne from comment #0)
> > > will this execute it between one jobs epilog and the next ones prolog once
> > > per hour , and if i have a job that say lasts for 16 hours , will it run it
> > > after the completion of the long 16 hour job  but before the next one.
> > The health_check is called by the cronlike _slurmctld_background() thread in
> > slurmctld. It then filters the node list from the nodes that are not DOWN.
> > There is no logic currently to ensure that it is executed between jobs. It
> > may execute between jobs but there is nothing making sure that it will
> > happen. Depending on how busy slurmctld is at the time, it is even less
> > likely to execute between jobs.
> 
> Please note that there is currently bug#12004 open to improving the
> functionality of health_check. I suggest taking a look.
> 
> If the goal is to ensure that the health_check runs after every job, then
> calling the health_check script via the epilog is the suggested route. It is
> already required that the health_check script request that the node gets
> closed so it should run there without any issues.

We have users that are running very short jobs.. , the node heath check runs longer than the jobs..  Also i am trying to remove/isolate admin tasks that can and on occasion do hang from the prolog and epilog so we dont get these  jobs getting hung in epilog or prolog for extended time. I am not as good as i would like to be to find jobs that are stuck in completing so i can run a scontrol reconfigure if issue is a commincation/munge cert timeout  issue similar to the prolog hangs. I will take a look

i Will look at that other bug you mentioned .

(In reply to mike coyne from comment #11)
> (In reply to Nate Rini from comment #10)
> > (In reply to Nate Rini from comment #4)
> > > (In reply to mike coyne from comment #0)
> > > > will this execute it between one jobs epilog and the next ones prolog once
> > > > per hour , and if i have a job that say lasts for 16 hours , will it run it
> > > > after the completion of the long 16 hour job  but before the next one.
> > > The health_check is called by the cronlike _slurmctld_background() thread in
> > > slurmctld. It then filters the node list from the nodes that are not DOWN.
> > > There is no logic currently to ensure that it is executed between jobs. It
> > > may execute between jobs but there is nothing making sure that it will
> > > happen. Depending on how busy slurmctld is at the time, it is even less
> > > likely to execute between jobs.
> > 
> > Please note that there is currently bug#12004 open to improving the
> > functionality of health_check. I suggest taking a look.
> > 
> > If the goal is to ensure that the health_check runs after every job, then
> > calling the health_check script via the epilog is the suggested route. It is
> > already required that the health_check script request that the node gets
> > closed so it should run there without any issues.
> 
> We have users that are running very short jobs.. , the node heath check runs
> longer than the jobs..  Also i am trying to remove/isolate admin tasks that
> can and on occasion do hang from the prolog and epilog so we dont get these 
> jobs getting hung in epilog or prolog for extended time. I am not as good as
> i would like to be to find jobs that are stuck in completing so i can run a
> scontrol reconfigure if issue is a communication/munge cert timeout  issue
> similar to the prolog hangs. I will take a look
> 
> i Will look at that other bug you mentioned .

It would be nice to have a config set able option for health check , and its time out when called from both the node health check and from the resume nodes script.. easier than caring a patch ..  i have not tried to port the change to the 22.05 with your new run_command functions quite yet,  as i was not sure the healthcheck and administrative commands like ansible runs  would function as we need as well as  avoiding running during a job to minimize jitter to running mpi jobs , but in a separate script( node health check ? )  so if it times out / hangs etc it will down the node and not hang up sequences of job launches on what may be functional hardware. 
Mike

Hey Mike -

I think I get where you're coming from here - you need something periodically handling health-checks on the node, but Epilog runs are too frequent, and NHC may interrupt work inadvertently due to the current de-coupling between job launches and the NHC runs.

I do think that's worth addressing, but I don't want to tackle it in isolation. It's become clear from discussions with a few labs that the current HealthCheck interfaces are inadequate. Bug 12004 has some other discussion related to this.

I'm unfortunately out of development time in the 23.02 release cycle, but would like to get something in place for 23.11 with the various interested parties. Are you / LANL willing to help review some design aspects of a new NHC integration (with some corresponding changes to the state management to avoid races such as the one described here) in a few months?

thanks,
- Tim

(In reply to Tim Wickberg from comment #15)
> Hey Mike -
> 
> I think I get where you're coming from here - you need something
> periodically handling health-checks on the node, but Epilog runs are too
> frequent, and NHC may interrupt work inadvertently due to the current
> de-coupling between job launches and the NHC runs.
> 
> I do think that's worth addressing, but I don't want to tackle it in
> isolation. It's become clear from discussions with a few labs that the
> current HealthCheck interfaces are inadequate. Bug 12004 has some other
> discussion related to this.
> 
> I'm unfortunately out of development time in the 23.02 release cycle, but
> would like to get something in place for 23.11 with the various interested
> parties. Are you / LANL willing to help review some design aspects of a new
> NHC integration (with some corresponding changes to the state management to
> avoid races such as the one described here) in a few months?
> 
> thanks,
> - Tim

Tim i will do what ever i can to help , i  will forward this on to my supervisor and see if i can get more traction. 
Mike

While I agree with Mike (and Matt) that the health check configuration in Slurm could benefit from some additional capabilities, I'll also note a few NHC features that can be helpful in these situations:

- Detached Mode (https://github.com/mej/nhc#detached-mode):  This allows NHC to fork() and return immediately to Slurm, avoiding any timeout issues.  Results from the backgrounded tests are returned on the *next* execution of NHC.  This is particularly useful when NHC is run at a (relatively) short interval but should *not* be used in Prolog/Epilog scripts (for obvious reasons). :-)

- Contexts:  If NHC is invoked with a unique context name (either on the command line, like "nhc -n nhc-slurm," or executed as a different name via a symlink or similar, e.g. "ln -sf nhc /usr/sbin/nhc-slurm" and "/usr/sbin/nhc-slurm"), an entirely different config file is read by default.  This context's config file (e.g., "/etc/nhc/nhc-slurm.conf") can have its own separate settings and everything.  (Note to @mcoyne:  I can certainly provide examples of this that I've used, but maybe the best example at this point is Graham's "CHC" work -- It's just NHC with a specific context!)

- Short-circuiting:  For situations where using Detached Mode is untenable for whatever reason, NHC's execution can be short-circuited inside its own config file; something like "check_cmd_status -r 1 pgrep -ax slurmstepd" as the 1st check will cause NHC to "fail" quickly if any slurmstepd processes are running.  Similar tricks can be done using "check_ps_unauth_users syslog die" (https://github.com/mej/nhc#check_ps_unauth_users) with careful configuration....

@mcoyne:  I'm happy to discuss options with you for temporary use until SchedMD is able to provide a more featureful health check interface. :-)

Any update on this effort ?

(In reply to mike coyne from comment #19)
> Any update on this effort ?

Hey Mike -

Nope, unfortunately. I do have some time now that 23.02 is out, and would love to pick this back up with some changes targeted for 23.11.

Let me see if I can get an overview written up of what I'd expect to change with some evolution of the NHC integration, and attach it here ~ next week for discussion.

@MEJ - has development for NHC picked back up in any capacity?

- Tim

(In reply to Tim Wickberg from comment #20)
> @MEJ - has development for NHC picked back up in any capacity?

Yes, substantially.  Now that I've (finally!) made it past all the legal nonsense and have full approval for myself and my teammates to contribute publicly, I'm doing quite a bit of work each week to get NHC moving forward again *and* to get the internal LANL development on NHC (it's been ongoing this whole time) merged in, genericized/tidied up, and pushed up to GitHub.  I'm also moving all my personal/private TODO tasks turned into GitHub Issues to make the roadmap more public as well.

Obviously, since I'm not 100% full-time dedicated to NHC development, it will still take some time to get caught up.  But I have total support from both my team and group leaders as well as the leadership of the Systems group.  Right now, as you know, pretty much all of HPC is swamped with Crossroads & Friends; however, I've already provided training sessions for several Platforms Team folks (and will continue to do so), and a number of new Shasta-focused checks have been assigned to other folks on the team (Alden, Steve Johnson, Travis Cotton, Graham, et al.) to develop and test.

I've also been asked by my superiors to come up with ways to promote NHC and "spread the word" that it's up and going again, which is quite promising in and of itself. :-)

So lots of good news on the NHC front!  And if there's anything I can do to help with or contribute to this effort, I'd be happy to!  I'm confident I'd have no trouble getting the go-ahead from both Mike's management and my own to help out with this.

is there any word on this ?

(In reply to mike coyne from comment #22)
> is there any word on this ?

waiting ?

Hey Mike -

I unfortunately haven't had any time to sit down and worth through this. I do think there's a lot of room for improvement, but have been putting out fires elsewhere for the past few releases.

I'm open to suggestions - if there's a clear set of requests for a modern interface from MEJ or others that's likely to be a bit easier for us to work through.

- Tim