Ticket 15305 - srun sigfault when SLURM_JOB_NODELIST is bracketed and terminated by a newline
Summary: srun sigfault when SLURM_JOB_NODELIST is bracketed and terminated by a newline
Status: RESOLVED FIXED
Alias: None
Product: Slurm
Classification: Unclassified
Component: User Commands (show other tickets)
Version: 22.05.3
Hardware: Linux Linux
: 4 - Minor Issue
Assignee: Chad Vizino
QA Contact:
URL:
Depends on:
Blocks:
 
Reported: 2022-10-28 00:51 MDT by Daniel Ahlin
Modified: 2023-02-20 11:42 MST (History)
1 user (show)

See Also:
Site: Google
Slinky Site: ---
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
Google sites: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
NoveTech Sites: ---
Nvidia HWinf-CS Sites: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Tzag Elita Sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed: 23.02.0rc2
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this ticket.
Description Daniel Ahlin 2022-10-28 00:51:06 MDT 
srun sigfaults when SLURM_JOB_NODELIST is bracketed and terminate by a newline.

Example:
$ salloc -N 2 -pc2
salloc: Granted job allocation 24
salloc: Waiting for resource configuration
salloc: Nodes ahlintopol-c2-ghpc-[0-1] are ready for job
$ echo $SLURM_NODELIST                 
ahlintopol-c2-ghpc-[0,1]
$ srun /usr/bin/hostname               
ahlintopol-c2-ghpc-0
ahlintopol-c2-ghpc-1
$ env SLURM_JOB_NODELIST="$SLURM_JOB_NODELIST
" srun /bin/hostname
Segmentation fault

Expectation:
While I agree it is an error that SLURM_JOB_NODELIST contains a newline I would expect the parsing to produce an error rather than sigfaulting

Impact:
The error is not all that easy to spot since e.g. echo'ing SLURM_JOB_NODELIST with and without the newline produces the same output (at least in bash). 

Diagnostics:
While the sigsegv is in hostlist_uniq, The issue is rather that the result of  hostlist_create at src/srun/libsrun/srun_job.c:293 is not checked.
Comment 1 Daniel Ahlin 2022-10-28 00:53:49 MDT 
After posting I noted that my example contains the legacy SLURM_NODELIST

$ echo $SLURM_NODELIST                 
ahlintopol-c2-ghpc-[0,1]

To be clear, SLURM_JOB_LIST contains the same:
echo $SLURM_JOB_NODELIST                 
ahlintopol-c2-ghpc-[0,1]
Comment 2 Chad Vizino 2022-10-28 14:07:08 MDT 
(In reply to Daniel Ahlin from comment #1)
> After posting I noted that my example contains the legacy SLURM_NODELIST
No problem with that env var. But indeed--this is a bug and easy to reproduce from what you have provided:

>$ salloc -N2
>salloc: Granted job allocation 12
>salloc: Waiting for resource configuration
>salloc: Nodes vizino-[3-4] are ready for job
>(salloc) $ env|grep NODEL
>(salloc) $ export SLURM_JOB_NODELIST="$SLURM_JOB_NODELIST
> > "
>(salloc) $ gdb --arg srun hostname
>...
>Starting program: /home/chad/slurm/master/vizino/bin/srun hostname
>...
>srun: error: ../../../../slurm/src/common/hostlist.c:2513: hostlist_uniq(): Assertion (hl != NULL) failed.
> 
>Program received signal SIGABRT, Aborted.
>...
>gdb) bt
>#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
>#1  __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
>#2  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
>#3  0x00007ffff783bc46 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
>#4  0x00007ffff78227fc in __GI_abort () at ./stdlib/abort.c:79
>#5  0x00007ffff7e73e45 in __xassert_failed (expr=0x7ffff7e8fe82 "hl != NULL", 
>    file=0x7ffff7e8fd88 "../../../../slurm/src/common/hostlist.c", line=2513, 
>    func=0x7ffff7e90718 <__func__.44> "hostlist_uniq") at ../../../../slurm/src/common/xassert.c:57
>#6  0x00007ffff7cf0f13 in hostlist_uniq (hl=0x0) at ../../../../slurm/src/common/hostlist.c:2513
>#7  0x000055555556808d in job_step_create_allocation (resp=0x5555555afb00, opt_local=0x55555557d0c0 <opt>)
>    at ../../../../../slurm/src/srun/libsrun/srun_job.c:294
>#8  0x000055555556ad61 in create_srun_job (p_job=0x55555557e0a8 <job>, got_alloc=0x7fffffffd54f, 
>    slurm_started=false, handle_signals=true) at ../../../../../slurm/src/srun/libsrun/srun_job.c:1348
>#9  0x000055555555d651 in srun (ac=2, av=0x7fffffffd6b8) at ../../../../slurm/src/srun/srun.c:193
>#10 0x0000555555560bd4 in main (argc=2, argv=0x7fffffffd6b8) at ../../../../slurm/src/srun/srun.wrapper.c:17
>(gdb)
Should be easy to fix and will look into it.
Comment 17 Chad Vizino 2023-02-20 11:42:25 MST 
Hi. This has been fixed in this commit:

>https://github.com/SchedMD/slurm/commit/0a614b8