Ticket 13531

Created attachment 23663 [details]
slurm.conf

Hello,

Following the Bug #13340, which is already resolved, and we could learn what was causing the issue, we're facing a similar problem with the "core file size" limits that are unlimited for Slurmd service, which occasionally causes the nodes to go to "Kill task failed" state.

A few months ago, we added the following "core file size" limits into the limits.conf file of every worker node (except login and head nodes) to prevent worker nodes from going to the "Kill task failed" state due to the following errors:

Jan 15 19:21:01 cpu-24-24 systemd[89824]: systemd-coredump@124-89751-0.service: Failed to set up network namespacing: No space left on device
Jan 15 19:21:01 cpu-24-24 systemd[89824]: systemd-coredump@124-89751-0.service: Failed at step NETWORK spawning /usr/lib/systemd/systemd-coredump: No space left on device

So, let's pick a reserved node:

[root@login-20-25 ~]# ssh cpu-25-8
[root@cpu-25-8 ~]# grep -v '#' /etc/security/limits.conf

* soft memlock unlimited
* hard memlock unlimited
* hard nofile 8192
* soft nofile 8192
* soft core 2097152
* hard core 4194304

> as you see, we added hard and soft limits for "core" (core file size) 

[root@cpu-25-8 ~]# ulimit -a
core file size          (blocks, -c) 2097152
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 2061717
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 8192
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 2061717
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

> The "ulimit" command confirms the changes

[root@cpu-25-8 ~]# ulimit -a user1
core file size          (blocks, -c) 2097152
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 2061717
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 8192
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 2061717
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

> Even a nonprivileged user has the correct "core file size" outside Slurm.

However, the limits of the slurmd process running inside the systemd looks totaly different:

[root@cpu-25-8 ~]# cat /proc/$(pidof slurmd)/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            unlimited            unlimited            bytes
Max core file size        unlimited            unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             2061717              2061717              processes
Max open files            131072               131072               files
Max locked memory         unlimited            unlimited            bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       2061717              2061717              signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us


So, let's login as a normal user to the login node, then make an interactive session to the same node as above:

$ ssh user1@login-20-25
[user1@login-20-25 ~]$ salloc -N1 -n 128 -p nocona --reservation=misha -w cpu-25-8
salloc: Granted job allocation 4953070
salloc: Waiting for resource configuration
salloc: Nodes cpu-25-8 are ready for job

[user1@cpu-25-8 ~]$ ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 2061717
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) 527826944
open files                      (-n) 8192
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 2061717
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

As you see here, the "core file size" is "unlimited, " the same as the ulimit settings on the login-20-25.
If I exit the interactive session and just do "srun" on the login node, I'll get the same:

[user1@cpu-25-8 ~]$ exit
[user1@login-20-25 ~]$ srun -n1 -N1 -p nocona --reservation=misha -w cpu-25-8 ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 2061717
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) 4123648
open files                      (-n) 8192
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 2061717
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Please also note the following setting on the worker nodes (such as cpu-25-8):

1) /etc/security/limits.d/ is empty
2) The settings in /etc/security/limits.conf:

* soft memlock unlimited
* hard memlock unlimited
* hard nofile 8192
* soft nofile 8192
* soft core 2097152
* hard core 4194304

3) There is no limit on user authentication at login under /etc/pam.d/
4) The Slurmd service file:

[root@cpu-25-8 ~]# cat /usr/lib/systemd/system/slurmd.service
[Unit]
Description=Slurm node daemon
After=munge.service network.target remote-fs.target
#ConditionPathExists=/etc/slurm/slurm.conf

[Service]
Type=simple
EnvironmentFile=-/etc/sysconfig/slurmd
ExecStart=/usr/sbin/slurmd -D $SLURMD_OPTIONS
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
LimitNOFILE=131072
LimitMEMLOCK=infinity
LimitSTACK=infinity
Delegate=yes
TasksMax=infinity

[Install]
WantedBy=multi-user.target

5) No limit has been set in /etc/systemd/* (We can look further if that's necessary!)
6) The "core file size" has already been excluded in slurm.conf from being propagated to the worker nodes:

# grep PropagateResourceLimitsExcept /etc/slurm/slurm.conf
PropagateResourceLimitsExcept=CORE,MEMLOCK,NPROC

So, I went ahead and stopped the slurmd service on cpu-25-8, and tried to run slurmd outside the systemd:

[root@cpu-25-8 ~]# systemctl stop slurmd
[root@cpu-25-8 ~]# slurmd -D &
[root@cpu-25-8 ~]# cat /proc/$(pidof slurmd)/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        4294967296           4294967296           bytes
Max resident set          unlimited            unlimited            bytes
Max processes             2061717              2061717              processes
Max open files            8192                 8192                 files
Max locked memory         unlimited            unlimited            bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       2061717              2061717              signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us


I also get a different ulimits inside an interactive session to cpu-25-8:

[user1@login-20-25 ~]$ salloc -N1 -n 128 -p nocona --reservation=misha -w cpu-25-8
salloc: Granted job allocation 4953070
salloc: Waiting for resource configuration
salloc: Nodes cpu-25-8 are ready for job

[user1@cpu-25-8 ~]$ ulimit -a
core file size          (blocks, -c) 4194304
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 2061717
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) 527826944
open files                      (-n) 8192
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 2061717
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

As you can see above, the "core file size" works correctly outside the systemd. I wondered whether you might have some idea on what possibly changes the ulimits of slurmd inside the systemd? Or, where should we look, and what is the possible solution for this?

Best Regards,
Misha