You can run mixed OS and cgroups mode on compute nodes. This should not have any impact on the slurmctld. Cgroups v1 is deprecated so there is no further bug fixes going into those features. If issues arise or of you choose to enable just v2 and the other nodes disabled then the same oom issues will be a concern in those nodes that do not have memory enforcement.  This could cause you multi node jobs to fail.

Hi Jason,

(In reply to Jason Booth from comment #1)
> You can run mixed OS and cgroups mode on compute nodes. This should not have
> any impact on the slurmctld. Cgroups v1 is deprecated so there is no further
> bug fixes going into those features. If issues arise or of you choose to
> enable just v2 and the other nodes disabled then the same oom issues will be
> a concern in those nodes that do not have memory enforcement.  This could
> cause you multi node jobs to fail.

Thanks for clarifying the mixed cgroups behavior on nodes and jobs!

We imagine that our jobs will never span partitions where nodes have different 
OS (EL8/EL9) and cgroups v1/v2 configurations.  Having OOMKillStep is another 
incentive to migrate nodes from Rocky 8 to Rocky 9.

Could you kindly update the OOMKillStep documentation [1] to state explicitly that 
this parameter only works on nodes where the three requirements are satisfied, 
and that non-compliant nodes will simply ignore the OOMKillStep configuration 
without causing any issues?

Best regards,
Ole

[1] https://slurm.schedmd.com/slurm.conf.html#OPT_OOMKillStep

Hello Ole,

Adding to what Jason stated, that is the general rule for cgroups, mixing them it is fine, as the rpc's from slurmctld are cgroup version agnostic, it is the slurmd that converts them to specific code for v1 or v2, that is why we can do the CgroupPlugin=autodetect in the cgroup.conf

The OOMKillStep is the same situation plus that if you run it with cgroup/v2 it works better. Let me explain:

For all nodes and cgroup versions it will detect a OOM in the end of the task and send a message to cancel the current step in all nodes.

But for nodes with cgroup/v2 and the memory.oom.group interface file in cgroups it will make that all the processes in the step are grouped together in case of oom, meaning that if a process in the job makes an oom all the processes in the job will get killed, thus triggering the oom kill process immediately.

I will update the documentation so that this is stated more clearly.

Best regards.

Hi Oriol,

(In reply to Oriol Vilarrubi from comment #3)
> Adding to what Jason stated, that is the general rule for cgroups, mixing
> them it is fine, as the rpc's from slurmctld are cgroup version agnostic, it
> is the slurmd that converts them to specific code for v1 or v2, that is why
> we can do the CgroupPlugin=autodetect in the cgroup.conf
> 
> The OOMKillStep is the same situation plus that if you run it with cgroup/v2
> it works better. Let me explain:
> 
> For all nodes and cgroup versions it will detect a OOM in the end of the
> task and send a message to cancel the current step in all nodes.
> 
> But for nodes with cgroup/v2 and the memory.oom.group interface file in
> cgroups it will make that all the processes in the step are grouped together
> in case of oom, meaning that if a process in the job makes an oom all the
> processes in the job will get killed, thus triggering the oom kill process
> immediately.

Thanks for confirming that OOMKillStep will work on both cgroup v1 and v2,
contrary to the documentation!

I have enabled TaskPluginParam=OOMKillStep in slurm.conf now since this seems to be a Good Thing.

> I will update the documentation so that this is stated more clearly.

Thanks, I think an update is strongly needed!

Best regards,
Ole

Hi Ole,

I have been looking deeper into this and found out that it was not working as expected the OOM detection and in consequence the OOM kill.

Only if the first task was the one causing an OOM, then the OOM was properly detected. So if the first task finished without OOM and then any other one caused an OOM that went undetected.

I am working on a patch to make the detection happen at the end of the step to ensure that all possible OOM events have already happened and also change the documentation to reflect this. With this change though the OOMKillStep work only on cgrop/v2, but it is better to work only for cgroup/v2 rather than to only work for the first task.

Long story short, we are not only fixing the documentation for OOMKillStep but also the code.

Regards.

Hi Oriol,

Thank you so much for looking deeper into this issue!

(In reply to Oriol Vilarrubi from comment #5)
> I have been looking deeper into this and found out that it was not working
> as expected the OOM detection and in consequence the OOM kill.
> 
> Only if the first task was the one causing an OOM, then the OOM was properly
> detected. So if the first task finished without OOM and then any other one
> caused an OOM that went undetected.
> 
> I am working on a patch to make the detection happen at the end of the step
> to ensure that all possible OOM events have already happened and also change
> the documentation to reflect this. With this change though the OOMKillStep
> work only on cgrop/v2, but it is better to work only for cgroup/v2 rather
> than to only work for the first task.
> 
> Long story short, we are not only fixing the documentation for OOMKillStep
> but also the code.

The main question is whether my parameter TaskPluginParam=OOMKillStep in slurm.conf
isn't causing any issues on our cgroup v1 (RockyLinux 8.10) nodes.  We have some 
Rocky 9.8 nodes where cgroup v2 hopefully will work.

Do you think I should disable TaskPluginParam=OOMKillStep at this time?

Best regards,
Ole

Hi Ole,

Having it enabled it has no problem with cgroup/v1 or v2, the difference is that with cgroup/v1 it will only work if the task that causes the OOM is the first one, whereas with cgroup/v2 and the appropiate cgroup interface memory.oom.group, that all rhel9 derivatives have, it will work with all the tasks.

The difference between cgroup/v1 and v2 is that in v1 it will work partially, that is all, so you are safe keeping the parameter set.

Regards.

Hi Oriol,

(In reply to Oriol Vilarrubi from comment #7)
> Having it enabled it has no problem with cgroup/v1 or v2, the difference is
> that with cgroup/v1 it will only work if the task that causes the OOM is the
> first one, whereas with cgroup/v2 and the appropiate cgroup interface
> memory.oom.group, that all rhel9 derivatives have, it will work with all the
> tasks.
> 
> The difference between cgroup/v1 and v2 is that in v1 it will work
> partially, that is all, so you are safe keeping the parameter set.

Thanks very much for this advice!  We will keep TaskPluginParam=OOMKillStep because
it's safe, although not very effective with cgroup v1.  

As we migrate our nodes to RockyLinux 9, the cgroup v2 will be used.

Best regards,
Ole