Ticket 15163 - cgroup_v2 plugin doesn't support swapaccount=0
Summary: cgroup_v2 plugin doesn't support swapaccount=0
Status: RESOLVED FIXED
Alias: None
Product: Slurm
Classification: Unclassified
Component: slurmd (show other tickets)
Version: 22.05.5
Hardware: Linux Linux
: 4 - Minor Issue
Assignee: Felip Moll
QA Contact:
URL:
Depends on:
Blocks:
 
Reported: 2022-10-12 15:31 MDT by Felix Abecassis
Modified: 2022-10-26 01:27 MDT (History)
0 users

See Also:
Site: NVIDIA (PSLA)
Slinky Site: ---
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
Google sites: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
NoveTech Sites: ---
Nvidia HWinf-CS Sites: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Tzag Elita Sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed: 22.05.6
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this ticket.
Description Felix Abecassis 2022-10-12 15:31:39 MDT 
We added swapaccount=0 to the kernel command-line on a node, see https://www.kernel.org/doc/html/v5.17/admin-guide/kernel-parameters.html for the documentation of this feature.

Running jobs still works, but there are 2 lines of error messages after each srun:

$ srun hostname
ioctl
slurmstepd-ioctl: error: Cannot read /sys/fs/cgroup/system.slice/ioctl_slurmstepd.scope/job_17/step_0/user/memory.swap.events
slurmstepd-ioctl: error: Cannot read /sys/fs/cgroup/system.slice/ioctl_slurmstepd.scope/job_17/step_0/user/memory.swap.events

Because those cgroupv2 files do not exist when using swapaccount=0.

However I then realized that this parameter just got deprecated for future Linux releases: https://github.com/torvalds/linux/commit/b25806dcd3d5248833f7d2544ee29a701735159f
So perhaps a documentation change would be enough, but silently ignoring those files should also be simple.
Comment 1 Felip Moll 2022-10-13 03:52:28 MDT 
(In reply to Felix Abecassis from comment #0)
> We added swapaccount=0 to the kernel command-line on a node, see
> https://www.kernel.org/doc/html/v5.17/admin-guide/kernel-parameters.html for
> the documentation of this feature.
> 
> Running jobs still works, but there are 2 lines of error messages after each
> srun:
> 
> $ srun hostname
> ioctl
> slurmstepd-ioctl: error: Cannot read
> /sys/fs/cgroup/system.slice/ioctl_slurmstepd.scope/job_17/step_0/user/memory.
> swap.events
> slurmstepd-ioctl: error: Cannot read
> /sys/fs/cgroup/system.slice/ioctl_slurmstepd.scope/job_17/step_0/user/memory.
> swap.events
> 
> Because those cgroupv2 files do not exist when using swapaccount=0.
> 
> However I then realized that this parameter just got deprecated for future
> Linux releases:
> https://github.com/torvalds/linux/commit/
> b25806dcd3d5248833f7d2544ee29a701735159f
> So perhaps a documentation change would be enough, but silently ignoring
> those files should also be simple.

Hi Felix, 

You're right, we are unconditionally reading the swap events file, and emitting an error (with no other real consequences) if that could not be read.

I will do some tests locally with swapaccount=0 and ensure we deal with this better.

Thanks for reporting.
Comment 7 Felip Moll 2022-10-26 01:27:24 MDT 
Felix,

This is fixed in commit 4438608b56, slurm 22.05.6.

NEWS: cgroup/v2 - Add check for swap when running OOM check after task termination.

Thanks!