Ticket 2493

Summary: slurmstepd segfaults error 4 in libc-2.12.so when TaskPlugin=task/cgroup RHEL 6.5
Product: Slurm Reporter: dani
Component: slurmstepdAssignee: Jacob Jenson <jacob>
Status: RESOLVED WONTFIX QA Contact:
Severity: 6 - No support contract    
Priority: ---    
Version: 15.08.8   
Hardware: Linux   
OS: Linux   
Site: -Other- Slinky Site: ---
Alineos Sites: --- Atos/Eviden Sites: ---
Confidential Site: --- Coreweave sites: ---
Cray Sites: --- DS9 clusters: ---
Google sites: --- HPCnow Sites: ---
HPE Sites: --- IBM Sites: ---
NOAA SIte: --- NoveTech Sites: ---
Nvidia HWinf-CS Sites: --- OCF Sites: ---
Recursion Pharma Sites: --- SFW Sites: ---
SNIC sites: --- Tzag Elita Sites: ---
Linux Distro: --- Machine Name:
CLE Version: Version Fixed:
Target Release: --- DevPrio: ---
Emory-Cloud Sites: ---

Description dani 2016-02-29 00:52:03 MST
Hi,

A very simple job causes a slurmstepd segfault and remains in running state indefinitely until canceled.

The exact error from dmesg:
slurmstepd[<pid>]: segfault at 0 ip 0000003da4f285a6 sp 00007fff29e02a08 error 4 in libc-2.12.so[3da4e00000+18b000]

This is reproducible on all nodes.

OS:
RHEL 6.5 stock (no updates)
glibc,kenel versions:
glibc-2.12-1.132.el6.x86_64
kernel: 2.6.32-431

slurm version 15.08.8

Slurm configuration combo that seems to trigger this behavior:
in slurm.conf
TaskPlugin=task/cgroup
in cgroup.conf
ConstrainDevices=yes

Other configuration parameters don't seem to have an effect on this behavior.

Our gres.conf file has the following format:
Name=ArbitraryString Count=int
Without attaching a device to a name (no file= attribute).


Our reference job:
cat cgroup.sbatch
#!/bin/bash
#SBATCH -N 1
#SBATCH --nodelist=n1
echo "Entering sleep"
sleep 5
echo "End"

Other symproms:
The freezer and other subsystems can't be cleared (with cgclear) due to "Device or resource busy"

This seems similar to https://bugs.schedmd.com/show_bug.cgi?id=1119

The segfault happens immediately - the output file is never written to (first echo never happens).
Comment 1 Jacob Jenson 2017-11-03 10:43:33 MDT
This version of Slurm is no longer supported.