Ticket 13950 - fix rss in sacct to include tmpfs and match memcg behaviour
Summary: fix rss in sacct to include tmpfs and match memcg behaviour
Status: OPEN
Alias: None
Product: Slurm
Classification: Unclassified
Component: Accounting (show other tickets)
Version: 21.08.7
Hardware: Linux Linux
: C - Contributions
Assignee: Tim Wickberg
QA Contact:
URL:
Depends on:
Blocks:
 
Reported: 2022-04-28 00:49 MDT by Robin Humble
Modified: 2022-04-28 10:11 MDT (History)
2 users (show)

See Also:
Site: -Other-
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
NoveTech Sites: ---
Nvidia HWinf-CS Sites: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Tzag Elita Sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed:
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---

Attachments
make accounting rss include tmpfs usage to match memcg behaviour (3.17 KB, patch)
2022-04-28 00:49 MDT, Robin Humble 		Details | Diff
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this ticket.
Description Robin Humble 2022-04-28 00:49:31 MDT 
Created attachment 24712 [details]
make accounting rss include tmpfs usage to match memcg behaviour

Hi,

we've been using a patch to fix rss in sacct from memcg for a few years.

the issue is that the kernel's memcg includes rss+tmpfs in its OOM decision making, but rss in sacct doesn't include tmpfs, so they don't always line up.

if jobs use a lot of shared mem or write plain files to /dev/shm, then jobs can be killed by OOM, but rss in sacct is wrong so it's confusing why the job was killed.

here's a patch to fix it.

TBH I kinda thought I'd submitted this patch a few years ago. hopefully this isn't a dup.

cheers,
robin