Damien, My understanding is that the cgroup.conf AllowedRAMSpace is set in the step's cgroup file:

memory.limit_in_bytes		 # set/show limit of memory usage

Then if AllowedSwapSpace is set another value is also set in this other step's cgroup file:

memory.memsw.limit_in_bytes	 # set/show limit of memory+Swap usage

Regarding what is accounted to determine memory usage, the kernel memory resource controller documentation[1] says that:

2.2.1 Accounting details

All mapped anon pages (RSS) and cache pages (Page Cache) are accounted.

[1] https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt

Hi Alex

Thanks for this information. 

In relations:

Recently, we have some troublesome jobs, (These jobs exceed their memory requirements in their #sbatch --mem-per-cpu ), these jobs have caused problems for other jobs.

At the current slurm.conf
------------
TaskPlugin=task/cgroup
------------

From what we observed, when it a certain ptreads use more memory(Rss) than requesed, cgroup will initialise OOM to kill this pthread.

Can We configure this to when Slurm detects that if  a certain job exceed it's memory request, it will initial slurmctl to kill/scancel this job instead ? Is this possible ?


Kindly advise. Thanks.


Cheers


Damien


(In reply to Alejandro Sanchez from comment #2)
> Damien, My understanding is that the cgroup.conf AllowedRAMSpace is set in
> the step's cgroup file:
> 
> memory.limit_in_bytes		 # set/show limit of memory usage
> 
> Then if AllowedSwapSpace is set another value is also set in this other
> step's cgroup file:
> 
> memory.memsw.limit_in_bytes	 # set/show limit of memory+Swap usage
> 
> Regarding what is accounted to determine memory usage, the kernel memory
> resource controller documentation[1] says that:
> 
> 2.2.1 Accounting details
> 
> All mapped anon pages (RSS) and cache pages (Page Cache) are accounted.
> 
> [1] https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt

With 'Task' plugin enabled, the oom killer takes care of step's memory limit. When step memory limit is exceeded the oom killer sends a SIGKILL signal to the process and Slurm marks the step as CANCELLED:

alex@pc:~/t$ srun --mem=1G memory_eater
slurmstepd-compute2: Exceeded step memory limit at some point.
srun: error: compute2: task 0: Killed
srun: Force Terminated job step 20001.0
alex@pc:~/t$ sacct -j 20001.0
       JobID    JobName  Partition    Account  AllocCPUS      State ExitCode 
------------ ---------- ---------- ---------- ---------- ---------- -------- 
20001.0      memory_ea+                    a1          2  CANCELLED      0:9 


With 'JobAcctGatherType' plugin enabled, step's memory is checked polling at a polling frequency specified by JobAcctGatherFrequency parameter. Again, if step's memory exceeds the limit, Slurm marks the step as CANCELLED:

alex@pc:~/t$ srun --mem=1G memory_eater
slurmstepd-compute2: Step 20003.0 exceeded memory limit (1126852 > 1048576), being killed
srun: Exceeded job memory limit
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
slurmstepd-compute2: *** STEP 20003.0 ON compute2 CANCELLED AT 2016-04-11T11:46:36 ***
srun: error: compute2: task 0: Killed
alex@pc:~/t$ sacct -j 20003.0
       JobID    JobName  Partition    Account  AllocCPUS      State ExitCode 
------------ ---------- ---------- ---------- ---------- ---------- -------- 
20003.0      memory_ea+                    a1          2 CANCELLED+      0:9 

If both 'task' and 'jobacctgathertype' plugins are enabled, probably the 'task' will make the step be killed as the oom killer detects faster the step's memory exceed than the polling interval configured with jobacctgatherfrequency.

So Slurm already kills the job step.

Hey Damien, can we also close this other bug? Thanks.

Hi Alex 

We have this scenario here in our site. We have already configure cgroups as mentioned in your ticket. And we have also the NHC installed.

When some jobs' memories exceeded, OOM did start killing of these processes, which is intended (Good ) but it also put these nodes off-line. 

We don't want these nodes to be off-line since SLURM has scancel these jobs, we want these nodes to be available for the next job coming in.


It might not be SLURM fault, as NHC detects this OOM kills, it listed them as kernel lockup faults and put these off-line.
------------------------
At 'kernel-lockups.nhc' from compute node:


count=$(grep -Ec "Killed process" $DMESGFILE)
if [ $count -ne 0 -a ! -e /usr/local/recluse/nodestatus/forcestatus/$(hostname)-oom ]
then
die 1 "$FUNCNAME ERROR kernel has killed processes due to OOM $moreinfo"
rm -f $DMESGFILE
return 1
fi

rm -f $DMESGFILE
----------------  

We wonder is there a way for SLURM to initalise OOM to kill these rogue processes without marking these nodes offline ? or we are asking the wrong question ?


Kindly advise. Thanks.


Damien

Damien, which HealthCheckProgram are you using? I can't find this message:
"ERROR kernel has killed processes due to"
in the LBNL Node Health Check[1]. Anyhow, I might be wrong but I think diagnosing node problems and changing the state of the nodes are responsibility of the healthcheckprogram being used. So in this case, my guess is that the dmesg messages parsing and the consequent actions being taken are not conducted by Slurm. I'm not very familiar with HealthCheckPrograms but taking a look at the LBNL implementation nhc.conf file you can setup this option:

# Do not mark nodes offline
#   * || export MARK_OFFLINE=0

Don't know if the one you are using have something similar or if the actions being taken when parsing oom messages can be modified.
 
[1] https://github.com/mej/nhc

I've just checked and Slurm itself doesn't explicitly set nodes to DOWN when oom killer takes action on an exceeded memory job:

alex@pc:~/t$ sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
part1*       up 1-03:00:00      2   idle compute[1-2]
part2        up    6:00:00      2   idle compute[1-2]
alex@pc:~/t$ srun -n1 --mem=50 memory_eater
slurmstepd-compute1: Step 20004.0 exceeded memory limit (615724 > 51200), being killed
srun: Exceeded job memory limit
slurmstepd-compute1: *** STEP 20004.0 ON compute1 CANCELLED AT 2016-04-13T18:03:19 ***
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
srun: error: compute1: task 0: Killed
alex@pc:~/t$ sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
part1*       up 1-03:00:00      2   idle compute[1-2]
part2        up    6:00:00      2   idle compute[1-2]
alex@pc:~/t$

Hi Alex

Yes, you are right, it is not SLURM's fault.

We are using NHC = Warewulf Node Health Check Script <mej@lbl.gov> 

I need to find a better method to do this.


Cheers

Damien



(In reply to Alejandro Sanchez from comment #8)
> I've just checked and Slurm itself doesn't explicitly set nodes to DOWN when
> oom killer takes action on an exceeded memory job:
> 
> alex@pc:~/t$ sinfo
> PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
> part1*       up 1-03:00:00      2   idle compute[1-2]
> part2        up    6:00:00      2   idle compute[1-2]
> alex@pc:~/t$ srun -n1 --mem=50 memory_eater
> slurmstepd-compute1: Step 20004.0 exceeded memory limit (615724 > 51200),
> being killed
> srun: Exceeded job memory limit
> slurmstepd-compute1: *** STEP 20004.0 ON compute1 CANCELLED AT
> 2016-04-13T18:03:19 ***
> srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
> srun: error: compute1: task 0: Killed
> alex@pc:~/t$ sinfo
> PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
> part1*       up 1-03:00:00      2   idle compute[1-2]
> part2        up    6:00:00      2   idle compute[1-2]
> alex@pc:~/t$

Damien, I'd try to modify the script spot where it detects oom killer messages to not scontrol the node DOWN. Is it fine to close the ticket?

Thanks.

Please close this ticket.


(In reply to Alejandro Sanchez from comment #10)
> Damien, I'd try to modify the script spot where it detects oom killer
> messages to not scontrol the node DOWN. Is it fine to close the ticket?

Marking as resolved.

Hi SchedMD Support.

I have this scenario (as related to this case)


---------------------------------
Show quoted text

#SBATCH --job-name=memory-checks
#SBATCH --account=pmosp
#SBATCH --ntasks=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=1


#SBATCH --mem-per-cpu=800

#SBATCH --time=00:05:00


cd /home/damienl/memtester-4.3.0
./memtester 1G 1
--------------------------------


[damienl@m2-login2 ~]$ sbatch mem-check-submit-script 
Submitted batch job 6143879

6143879 m2-com-c6 memory-c damienl R 0:00 1 m2048


[damienl@m2-login2 ~]$ cat slurm-6143879.out 
memtester version 4.3.0 (64-bit)
Copyright (C) 2001-2012 Charles Cazabon.
Licensed under the GNU General Public License version 2 (only).
pagesize is 4096
pagesizemask is 0xfffffffffffff000
want 1024MB (1073741824 bytes)
got 1024MB (1073741824 bytes), trying mlock ...locked.
Loop 1/1:
Stuck Address : setting 2slurmstepd: Job 6143879 exceeded memory limit (1050276 > 819200), being killed
slurmstepd: Exceeded job memory limit
slurmstepd: *** JOB 6143879 CANCELLED AT 2016-04-28T16:11:17 *** on m2048
[damienl@m2-login2 ~]$ 


From node m2048
dmesg: show some messages but not related to 6143879

slurmd
[2016-04-28T16:11:17.531] [6143879] Job 6143879 exceeded memory limit (1050276 > 819200), being killed
[2016-04-28T16:11:17.545] [6143879] Sent signal 18 to 6143879.4294967294
[2016-04-28T16:11:17.556] send notification to job 6143879.4294967294
[2016-04-28T16:11:17.556] [6143879] Exceeded job memory limit
[2016-04-28T16:11:17.556] [6143879] *** JOB 6143879 CANCELLED AT 2016-04-28T16:11:17 *** on m2048
[2016-04-28T16:11:17.556] [6143879] task 0 (19816) exited. Killed by signal 15.
[2016-04-28T16:11:17.557] [6143879] Sent signal 15 to 6143879.4294967294
[2016-04-28T16:11:18.565] [6143879] job 6143879 completed with slurm_rc = 0, job_rc = 15
[2016-04-28T16:11:18.565] [6143879] sending REQUEST_COMPLETE_BATCH_SCRIPT, error:0 status 15
[2016-04-28T16:11:18.566] [6143879] done with job



This shows the slurmd actually kill this job (which is correct), although I have configure cgroups, I don't see any OOM kills from syslogs or dmesg.

I just wanted to know if this behaviour is in order.


Kindly advise. Thanks.

Cheers

Damien

I've submitted your same script and OOM killer comes into play in dmesg:

alex@pc:~/t$ scontrol show config | grep -Ei "cgroup|version"
JobAcctGatherType       = jobacct_gather/cgroup
ProctrackType           = proctrack/cgroup
SLURM_VERSION           = 14.11.3
TaskPlugin              = task/cgroup
alex@pc:~/t$ cat script.batch 
#!/bin/bash
#SBATCH --job-name=memory-checks
#SBATCH --account=alex
#SBATCH --ntasks=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=200
#SBATCH --time=00:05:00
srun memory_eater
alex@pc:~/t$ sbatch script.batch 
Submitted batch job 20004
alex@pc:~/t$

From dmesg:
Apr 28 11:22:55 pc kernel: [ 2815.324200] memory_eater invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
Apr 28 11:22:55 pc kernel: [ 2815.324453] Memory cgroup out of memory: Kill process 6693 (memory_eater) score 990 or sacrifice child
Apr 28 11:22:55 pc kernel: [ 2815.324456] Killed process 6693 (memory_eater) total-vm:209012kB, anon-rss:201268kB, file-rss:1256kB

Would you mind double-checking you've no "Killed process" messages in your dmesg output?

Hi Damien, have you had the chance to double check the log messages? Thanks.

Hello Damien. Any updates on this?

Hi Alex

Sorry for this late reply.

I figure out that we have configured our Cent)S6.5 cluster's cgroups conf wrongly, this is no fault on SLURMS.

We are still trying to configure cgroups properly in our Cluster.


Cheers

Damien

Hi Damien. Sure,  please let me know once you have cgroups properly configured in your cluster.

Hi Damien. Any progress with this bug? Did you manage to setup cgroups on your CentOS 6.5 cluster? Thanks.

Ping. Any progress with the cgroups con CentOS 6.5?

Hi Damien. Since I didn't hear any more issues since May and in comment #17 you mentioned that it was not a Slurm fault and that you had to properly configur cgroups in your CentOS 6.5 cluster, I'm gonna proceed to close the bug. Please, do not hesitate in reopening if you face any more issues. Thanks.

Hi Alex

I was not in office last week.

Yes, we are updating our SLURM version soon. Please close this ticket. Thanks.


Cheers

Damien

All right, closing the bug. Thanks Damien.