Ticket 20468 - RFE: do not immediately SIGKILL all processes in a task when the main process exits
Summary: RFE: do not immediately SIGKILL all processes in a task when the main process...
Status: RESOLVED FIXED
Alias: None
Product: Slurm
Classification: Unclassified
Component: slurmstepd (show other tickets)
Version: 24.11.x
Hardware: Linux Linux
: 5 - Enhancement
Assignee: Ben Glines
QA Contact:
URL:
Depends on:
Blocks:
 
Reported: 2024-07-22 19:12 MDT by Felix Abecassis
Modified: 2025-09-02 09:30 MDT (History)
5 users (show)

See Also:
Site: NVIDIA (PSLA)
Slinky Site: ---
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
Google sites: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
NoveTech Sites: ---
Nvidia HWinf-CS Sites: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Tzag Elita Sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed: 25.05.0rc1
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this ticket.
Description Felix Abecassis 2024-07-22 19:12:15 MDT 
From what I have noticed experimentally and also by looking at the code in src/slurmctld/job_mgr.c, Slurm will immediately send a SIGKILL to all remaining processes when all the tasks on a node are considered finished. And a task is considered finished when the main PID, the one that was directly spawned by slurmstepd, exits.

$ time srun bash -c 'bash -c "sleep 21 & kill -9 \$$" ; sleep 3s ; pgrep -a sleep'
/usr/bin/bash: line 1: 710924 Killed                  bash -c "sleep 21 & kill -9 \$$"
710907 sleep 100000000
710925 sleep 21

real    0m4.149s
user    0m0.003s
sys     0m0.003s

And from the slurmstepd log:
[1342.0] debug2: proctrack/cgroup: proctrack_p_signal: sending process 710925 (inherited_task) signal 9


We have a few use cases where some of the subprocesses of the task serve an important purpose (for example in a "sidecar container" type of pattern) and we would like the descendants of the task to not be terminated so forcefully, to give them a chance to cleanup and also to notify other nodes in the job that the application terminated.

There are probably a few ways to enable to this feature, for example I could think of the following:
- There could be a configuration option similar in behavior to GraceTime, but for subprocesses in a task: they would be sent SIGTERM and then SIGKILL after N seconds.

- There could be a new "srun" flag that waits until *all* processes in a task exit before marking the task as complete. I looked at this and it seems more challenging to implement, you need PR_SET_CHILD_SUBREAPER to be able to reap grandchildren processes (and avoid having them reparented to init), but when you reap a reparented process you need to be able to lookup which local task it belonged to before it was reparented to slurmstepd.

We are open to other suggestions, of course.
Comment 1 Tim Wickberg 2024-07-23 17:15:50 MDT 
Re-tagging as a potential enhancement request.

Is this something you're imagining only needing on explicitly launched steps with srun, or might also want as part of the batch step as well?

Using PR_SET_CHILD_SUBREAPER is certainly trickier, but also may be worth exploring on its own anyways.
Comment 2 Felix Abecassis 2024-07-23 18:08:52 MDT 
Right now our use case is only for steps launched through srun. But we believe supporting processes in the sbatch test could possibly be helpful too, and the behavior would be consistent this way between the sbatch step and srun steps.
Comment 3 Ben Glines 2025-05-02 16:41:38 MDT 
The '--wait-for-children' options has been added to srun and will be available in 25.05. The option leverages certain cgroup features that make it possible to wait until all processes are done before tasks (and subsequently the step) finished.

Please let me know if you have any questions.

See commits here: https://github.com/SchedMD/slurm/compare/f3a3d87f3feb...0a25b5ffd297.
Comment 4 Ben Glines 2025-05-09 09:56:06 MDT 
Please reopen this ticket or submit a new ticket if you run into any issues. Closing now.
Comment 5 Felix Abecassis 2025-06-26 17:13:18 MDT 
Re-opening, as I would like to discuss this aspect of the feature: https://slurm.schedmd.com/srun.html#OPT_wait-for-children

> Note that if the parent process exits with a non-zero exit code, the task will end regardless of whether there are still children processes running.

After discussing with our internal users that requested this feature, we would like to discuss whether this constraint can be relaxed.

The use case is having children processes monitor the parent process and give those children the time to notify other nodes that one task crashed, so they should not be killed immediately.

The suggestion from a user, and I think it makes sense, is to have a special case when both --no-kill and --wait-for-children are set. In this case we would wait wait all for children processes even if the exit code was non-zero for the parent.

If --no-kill is not set, then we would keep the current behavior.

What do you think?
Comment 6 Ben Glines 2025-07-03 15:44:33 MDT 
Hi Felix,

We're currently discussing this internally. I'll get back to you ASAP on how we want to handle this.
Comment 7 Julie Bernauer 2025-07-03 15:45:07 MDT 
Traveling. Email replies will be delayed.
Comment 9 Tim Wickberg 2025-07-07 15:29:33 MDT 
Hey Felix -

I'm not sure I want to overload --no-kill in this way, that flag has a lot of other impacts that aren't necessarily directly tied here.

It looks like --kill-on-bad-exit=0 would be suitable. Although on my quick testing it seems like that's only respected client-side right now, and we'd need to adjust some code paths to suit. I'll ask Ben to take a look at whether we could patch that in, or if this might also require some RPC changes to support.

- Tim
Comment 11 Felix Abecassis 2025-08-01 10:11:44 MDT 
Hello Tim,

Do you have any update on this? Regarding modifying the behavior when --kill-on-bad-exit=0 is used.

Thanks
Comment 12 Ben Glines 2025-08-04 16:01:47 MDT 
(In reply to Felix Abecassis from comment #11)
> Hello Tim,
> 
> Do you have any update on this? Regarding modifying the behavior when
> --kill-on-bad-exit=0 is used.
> 
> Thanks

We have a patch set to use --kill-bad-on-exit to change the behavior of --wait-for-all-children in regards to the main process exit, i.e. if --kill-on-bad-exit=0, ignore non-zero exit code from main process, and if --kill-on-bad-exit=1, task is ended when main process ends with a non-zero exit code. The patch set is currently under review and I'll let you know about any status updates on it.

Please let me know if you have any questions or concerns
Comment 13 Felix Abecassis 2025-08-05 08:41:19 MDT 
Thanks Ben!

Are you planning to add the patchset to 25.05 or just to master/25.11?
Comment 14 Ben Glines 2025-08-05 09:11:01 MDT 
(In reply to Felix Abecassis from comment #13)
> Thanks Ben!
> 
> Are you planning to add the patchset to 25.05 or just to master/25.11?

We are considering adding it to 25.05 as well, but I can't guarantee that yet. I'll let you know what is decided during the review process.
Comment 15 Ben Glines 2025-08-05 14:33:10 MDT 
We've added this change to --wait-for-children to 25.11, as well as 25.05. The earliest 25.05 version to see this change should be 25.05.2. See the following commits for each respective version:

25.11:
https://github.com/SchedMD/slurm/compare/d1ebc21f15b...5f191368bb7

25.05:
https://github.com/SchedMD/slurm/compare/d0bb9b329ef...66289192195

Let me know if you have any questions.
Comment 16 Ben Glines 2025-09-02 09:30:02 MDT 
Closing.