Ticket 8140

Tommi,

Can you please attach your slurmd log of the node where you tested this?

Thanks,
--Nate

Created attachment 12395 [details]
slurmd log

Test which I ran:

[ttervo@c1 ~]$ srun -n1 -p small --mem-per-cpu=100k --pty $SHELL
[ttervo@c1 ~]$ ./memtest 
Enter number of int(4 byte) you want to allocate:360000000
Allocating 1440000000 bytes......
Filling int into memory.....
Sleep 60 seconds......
Free memory.

(In reply to Tommi Tervo from comment #0)
> JobAcctGatherParams     = UsePss,OverMemoryKill
> AccountingStorageEnforce = associations,limits,qos

Please also set "MemLimitEnforce=yes" in your slurm.conf.

----- bugs@schedmd.com wrote:
> https://bugs.schedmd.com/show_bug.cgi?id=8140
> 
> --- Comment #3 from Nate Rini <nate@schedmd.com> ---

> Please also set "MemLimitEnforce=yes" in your slurm.conf.
> 
 
Like I wrote, it  is deprecated/removed from 19.05. It is on the config file but ignored.

[root@slurmctl ~]# grep -i memlim /etc/slurm/slurm.conf
MemLimitEnforce=YES
[root@slurmctl ~]# systemctl restart slurmctld
[root@slurmctl ~]# scontrol show config |grep -i memlim
[root@slurmctl ~]# echo $?
1

Is this a Cray Aries cluster?

(In reply to Nate Rini from comment #7)
> Is this a Cray Aries cluster?

No, Centos7 cluster.

-Tommi

Tommi

I believe I have confirmed the bug. I will work on a patchset.

Thanks,
--Nate

Tommi

Does this node have a swap enabled?
> cat /proc/swaps 

Thanks,
--Nate

> Does this node have a swap enabled?
> > cat /proc/swaps 

Hi,

Yes it has swap but swappiness seems to be zero:

[ttervo@c1 ~]$ cat /proc/sys/vm/swappiness 
0

[ttervo@c1 ~]$ free
              total        used        free      shared  buff/cache   available
Mem:        2046892      165196     1705348       23160      176348     1706428
Swap:       1048572        2560     1046012
[ttervo@c1 ~]$ cat /proc/swaps 
Filename                                Type            Size    Used    Priority
/dev/dm-1                               partition       1048572 2560    -2

(In reply to Tommi Tervo from comment #13)
> Yes it has swap but swappiness seems to be zero:
> [ttervo@c1 ~]$ cat /proc/swaps 
> Used=2560

Looks like it is still getting used. In my testing, I found that the rlimit was set on the process and all the memory above the requested allocation went to swap.  Since swapped out pages don't count against the memory RSS usage, Slurm was not killing the processes.

Is it possible to call this and try again?
> swapoff /dev/dm-1

Thanks,
--Nate

> Is it possible to call this and try again?
> > swapoff /dev/dm-1

Hi,

It did not have any effect:

[root@c1 ~]# swapoff -a
[root@c1 ~]# free
              total        used        free      shared  buff/cache   available
Mem:        2046892      173868     1681852       33400      191172     1685220
Swap:             0           0           0
[root@c1 ~]# logout
[ttervo@c1 ~]$ srun -n1 -p small --mem-per-cpu=100k --pty $SHELL

[ttervo@c1 ~]$ ./memtest
Enter number of int(4 byte) you want to allocate:360000000
Allocating 1440000000 bytes......
Filling int into memory.....
Sleep 60 seconds......
Free memory.
[ttervo@c1 ~]$ exit
exit
[ttervo@c1 ~]$ sacct -j 53 -oreqmem,maxrss
    ReqMem     MaxRSS 
---------- ---------- 
       1Mc            
       1Mc          0 
       1Mc   1407896K

(In reply to Tommi Tervo from comment #15)
> > Is it possible to call this and try again?
> > > swapoff /dev/dm-1
> It did not have any effect:

Thanks for verifying. Working on a patch set now.

Tommi,

A patch is undergoing review, please tell me if you need it soon than the normal review process allows.

Thanks,
--Nate

(In reply to Nate Rini from comment #24)
> Tommi,
> 
> A patch is undergoing review, please tell me if you need it soon than the
> normal review process allows.


Hi,

I could apply it to my test environment for additional testing.

Thanks,
Tommi

(In reply to Nate Rini from comment #22)
> Created attachment 12526 [details]
> patch

Please give it a try on your test system.

Created attachment 12544 [details]
slurmd log with patch 12526

Hi,

I've a bad news, patch did not help. I verified that build is using patched source code:

[root@slurmctl slurm-19.05.4]# grep -A1 'clone slurmctld config' /root/rpmbuild/BUILD/slurm-19.05.4/src/slurmd/slurmd/slurmd.c
        /* clone slurmctld config into slurmd config */
        conf->job_acct_oom_kill = slurmctld_conf.job_acct_oom_kill;

[ttervo@c1 ~]$ srun -n1 -p small --mem-per-cpu=100k --pty $SHELL
[ttervo@c1 ~]$ ./memtest 
Enter number of int(4 byte) you want to allocate:360000000
Allocating 1440000000 bytes......
Filling int into memory.....
Sleep 60 seconds......
Free memory.
[ttervo@c1 ~]$ exit
[ttervo@c1 ~]$ sacct -j 56 -o maxrss,reqmem
    MaxRSS     ReqMem 
---------- ---------- 
                  1Mc 
      394K        1Mc 
  1408230K        1Mc

Tommi,

Can you please verify that the slurmd daemon was fully restarted on the test node?
> [2019-12-12T10:15:24.354] error: Error binding slurm stream socket: Address already in use
> [2019-12-12T10:15:24.354] error: Unable to bind listen port (*:6818): Address already in use

Can you please verify that the cgroups are mounted on the node in the expected location?
> 2019-12-12T10:15:47.202] [55.0] debug2: xcgroup_load: unable to get cgroup '/sys/fs/cgroup/cpuset' entry '/sys/fs/cgroup/cpuset/slurm/system' properties: No such file or directory
> [2019-12-12T10:15:47.202] [55.0] debug2: xcgroup_load: unable to get cgroup '/sys/fs/cgroup/memory' entry '/sys/fs/cgroup/memory/slurm/system' properties: No such file or directory

While the job is sleeping for 60 seconds, can you please call gcore against the attached slurmstepd and provide 't a a bt full' output from gdb?

Thanks,
--Nate

(In reply to Nate Rini from comment #28)
> Tommi,
> 
> Can you please verify that the slurmd daemon was fully restarted on the test
> node?
> > [2019-12-12T10:15:24.354] error: Error binding slurm stream socket: Address already in use
> > [2019-12-12T10:15:24.354] error: Unable to bind listen port (*:6818): Address already in use


Doh, seems that systemctl could not stop old slurmd and I did not catch that from verbose log, looked only for updated slurmd version string. After kill -9 `pidof slurmd` and systemctl start slurmd overmemorykill works fine on my test system. Thanks for fix.

Best Regards,
Tommi Tervo
CSC

Tommi,

This is now fixed upstream by 4edf4a5898a2944. Please reply if you have any questions or issues.

Thanks,
--Nate