Ticket 2493 - slurmstepd segfaults error 4 in libc-2.12.so when TaskPlugin=task/cgroup RHEL 6.5
Summary: slurmstepd segfaults error 4 in libc-2.12.so when TaskPlugin=task/cgroup RHEL...
Status: RESOLVED WONTFIX
Alias: None
Product: Slurm
Classification: Unclassified
Component: slurmstepd (show other tickets)
Version: 15.08.8
Hardware: Linux Linux
: 6 - No support contract
Assignee: Jacob Jenson
QA Contact:
URL:
Depends on:
Blocks:
 
Reported: 2016-02-29 00:52 MST by dani
Modified: 2017-11-03 10:43 MDT (History)
0 users

See Also:
Site: -Other-
Slinky Site: ---
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
Google sites: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
NoveTech Sites: ---
Nvidia HWinf-CS Sites: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Tzag Elita Sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed:
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this ticket.
Description dani 2016-02-29 00:52:03 MST 
Hi,

A very simple job causes a slurmstepd segfault and remains in running state indefinitely until canceled.

The exact error from dmesg:
slurmstepd[<pid>]: segfault at 0 ip 0000003da4f285a6 sp 00007fff29e02a08 error 4 in libc-2.12.so[3da4e00000+18b000]

This is reproducible on all nodes.

OS:
RHEL 6.5 stock (no updates)
glibc,kenel versions:
glibc-2.12-1.132.el6.x86_64
kernel: 2.6.32-431

slurm version 15.08.8

Slurm configuration combo that seems to trigger this behavior:
in slurm.conf
TaskPlugin=task/cgroup
in cgroup.conf
ConstrainDevices=yes

Other configuration parameters don't seem to have an effect on this behavior.

Our gres.conf file has the following format:
Name=ArbitraryString Count=int
Without attaching a device to a name (no file= attribute).


Our reference job:
cat cgroup.sbatch
#!/bin/bash
#SBATCH -N 1
#SBATCH --nodelist=n1
echo "Entering sleep"
sleep 5
echo "End"

Other symproms:
The freezer and other subsystems can't be cleared (with cgclear) due to "Device or resource busy"

This seems similar to https://bugs.schedmd.com/show_bug.cgi?id=1119

The segfault happens immediately - the output file is never written to (first echo never happens).
Comment 1 Jacob Jenson 2017-11-03 10:43:33 MDT 
This version of Slurm is no longer supported.