(In reply to Felix Abecassis from comment #0)
> It gets even nastier when combining this bug with a node_features plugin and
> a constraint requiring a reboot. Simultaneously, the node starts rebooting
> and the job is cancelled. I suppose Slurm lost track of the reboot request
> since the node is immediately drained with the following:
The combination of this bug with our node_features plugin[s] is indeed nasty, but it's not precisely because Slurm loses track of the reboot request.

The issue that I've observed is this:

1. Slurm recognizes that a reboot is required in order to fulfill the constraint of the job. It calls node_features_p_node_set() for our plugin before rebooting the node.
2. Slurm recognizes that the srun process has been killed and that the job should be cancelled. It kicks off the epilog (why?). Bizarrely, the prolog never runs, just the epilog.
3. There is a race condition between these two tasks.

Sometimes, the plugin finishes node_features_p_node_set() and then initiates the reboot before the epilog starts. In this case, the epilog fails with all sorts of crazy errors because the node is in the process of shutting down.

Sometimes, the epilog runs while the node_features_p_node_set() function is still running. In one case, the plugin uninstalls and reinstalls the nvidia driver. Depending on which task happens first, sometimes this causes the epilog to fail. Sometimes it causes the driver reinstall to fail (which is the worst of all, because the node reboots without a driver installed).

Here are the relevant snippets of the slurmctld and slurmd logs for one of these  failure modes:

> Mar 22 16:42:07 login-node slurmctld[11735]: JobId=98975 nhosts:1 ncpus:1 node_req:64000 nodes=compute-node-747
> Mar 22 16:42:07 login-node slurmctld[11735]: reboot_job_nodes: reboot nodes compute-node-747 features nvidia_driver=XXX
> Mar 22 16:42:07 login-node slurmctld[11735]: sched/backfill: _start_job: Started JobId=98975 in batch on compute-node-747
> Mar 22 16:42:07 compute-node-747 slurmd[3840]: Node reboot request with features nvidia_driver=XXX being processed
> Mar 22 16:42:07 compute-node-747 systemd[1]: Stopping LSB: Startup/shutdown script for GDRcopy kernel-mode driver...
> ...
> Mar 22 16:42:18 compute-node-747 slurm[8592]: Running /etc/slurm/epilog.d/50-exclusive-gpu ...
> Mar 22 16:42:18 compute-node-747 slurm[8623]: Draining node -- Failed to reset GPU power levels
> Mar 22 16:42:18 login-node slurmctld[11735]: update_node: node compute-node-747 reason set to: Failed to reset GPU power levels
> Mar 22 16:42:18 login-node slurmctld[11735]: update_node: node compute-node-747 state set to DRAINED$

So, there are two bug reports here:

1. Killing the srun should delete the job from the queue to avoid scheduling issues. This can happen at any site and is not related to node_features plugins. It's bad from a cluster occupancy point of view, but at least it cleans up nicely.

2. There is a race condition between rebooting a node for a node_features plugin and running the epilog for a cancelled job. This has horrendous implications for our site because it can leave the node in a broken state. But it only applies for sites which use node_features plugins, as far as I can see.

Looking through both of your comments, I agree with Luke's statement that there are really 2 issues here.

Would you mind splitting the node_features plugin + epilog related issue into a separate ticket just to keep the histories clean?  I'll look into what is happening with srun in this ticket.

I was able to reproduce the behavior with srun, I'm currently looking into exactly what is happening here and why.

Thanks!
--Tim

(In reply to Tim McMullan from comment #3)
> Would you mind splitting the node_features plugin + epilog related issue
> into a separate ticket just to keep the histories clean?  I'll look into
> what is happening with srun in this ticket.
Sure, no problem. Bug#11182.

Answering here on a comment raised in https://bugs.schedmd.com/show_bug.cgi?id=11182

> So just avoid sending SIGKILL to an srun and you should be good to go.

So, SIGKILL was an example, it can happen in other situations. And even if it's indeed a SIGKILL, we can't prevent our users from sending SIGKILL to their own processes. So it would be great if we the job could be cancelled earlier than it is today.

I've been able to reproduce this and I've been looking around in the code surrounding it.

There is something that is related to this idea that is controlled by the InactiveLimit" in slurm.conf, however it is only meant for running jobs and something totally new would likely need to be made to do this for scheduled jobs.  I don't think setting the InactiveLimit would help in your case either, since its discovering that the srun died due to failed I/O basically immediately.

I think at this time checking a scheduled but not allocated srun would likely be an enhancement, but I am going to discuss this more internally before I can say that for sure.

Thanks!
--Tim

Thanks Tim!

Would it be easier to just patch the Stdout of the job?
e.g. setting Stdout/Stderr to the default of "slurm-%j.out", instead of cancelling the job?

That might be possible to do, but it really sounds to me like sbatch might just be the right solution.  Using srun directly certainly works, but a job that could take days to schedule and could be output to a file feels like it should just be a batch job.  sbatch has a "--wrap" option which can make life a little easier when just running a single command as a batch script too.

Is there something that makes sbatch not work for this?

Thanks!
--Tim

> That might be possible to do, but it really sounds to me like sbatch might just be the right solution.  Using srun directly certainly works, but a job that could take days to schedule and could be output to a file feels like it should just be a batch job. 

I agree, and we do recommend that our users use sbatch for most of their workloads. But there are some use cases where srun is useful, and we can't prevent users from using srun for a large job (can we?).

What I'm suggesting is effectively an automatic "promotion" from an interactive (non-pty) srun to a batch job, only for those jobs that we know are going to fail immediately, since the receiving process is gone. This will avoid wasting resources by having the job run to completion. 

I figured that patching the job when it starts might be simpler than figuring out in advance when the job can be safely cancelled, as initially requested.

Note: I would still prefer to have the job cancelled early instead of being promoted to a batch job when it is being started.

I can see the utility in something like this, but right now its operating as intended, so I think it would be an enhancement to change the behavior.

Preventing the user from submitting a large srun outside of an allocation is an interesting idea, and it may be possible using a cli filter plugin.  With cli_filter/lua you can detect that it was srun with 'args.type == "srun"', then check if you are in an allocation by checking if "SLURM_JOB_ID" exists in the users environment.  If it doesn't, you could reject an srun if its requesting 90 nodes before it ever gets submitted... but allow an sbatch to get scheduled normally.

Would something like that work?

Thanks!
--Tim

> I can see the utility in something like this, but right now its operating as intended, so I think it would be an enhancement to change the behavior.

Could you clarify which part is working as intended?
For now, let's disregard the suggestion to inject a Stdout path for these jobs, do we agree that the behavior reported initially in this thread is a bug?


> [...]
> Would something like that work?

It could work fine as a stopgap measure, but we already have a good amount of logic in our job_submit Lua script; hence we would prefer to have a fix in Slurm instead.

I'm sorry about the delay on this, but I did some chatting internally about what we wanted to do on this.

We are going to consider this an enhancement for now, particularly since any change we might make here would have to wait for a new stable release.

I am taking a look at what exactly would be required to alter the behavior here though!

Thanks,
--Tim

Thanks for the update, Tim!

Following our meeting today, we would like to prevent slurmd from running the prolog and if possible also prevent slurmd from rebooting the node (for a node feature change).

We understand that we don't want slurmctld to continually monitor the state of the client processes.