Ticket 19912 - MaxRss calculation in Accounting
Summary: MaxRss calculation in Accounting
Status: OPEN
Alias: None
Product: Slurm
Classification: Unclassified
Component: Accounting (show other tickets)
Version: - Unsupported Older Versions
Hardware: Linux Linux
: 6 - No support contract
Assignee: Jacob Jenson
QA Contact:
URL:
Depends on:
Blocks:
 
Reported: 2024-05-17 04:47 MDT by j.naumann
Modified: 2024-05-17 04:47 MDT (History)
0 users

See Also:
Site: -Other-
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
NoveTech Sites: ---
Nvidia HWinf-CS Sites: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Tzag Elita Sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed:
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---


Attachments

Note You need to log in before you can comment on or make changes to this ticket.
Description j.naumann 2024-05-17 04:47:46 MDT
Dear slurm support,

I encountered a weird behavior about the MaxRss field in the slurm account. For the concrete example sacct reports [1] a MaxRss of roughly 55.7G while the job had a mem limit of 32G. If one measures the MaxRss of the job script outside of slurm with the GNU time-tool (/usr/bin/time -v ...) you get a MaxRss of 28G which is about the half of the size sacct reports.  

Therefore, I want to ask you how the MaxRss field of sacct is calculated? Is there a factor inside the slurm value which one has to consider? How can the fact be explained that slurm reports a MaxRss of 56G which would be clear above the mem limit of 32G which is enforced by the slurm cgroups plugin?

Thank you very much!
Jan Naumann 

[1]

$ sacct  -j 273186_0 -o JobId,MaxVMSize,MaxVMSizeNode,MaxVMSizeNode,AveVMSize,MaxRSS,MaxRSSNode,MaxRSSTask,AveRSS,AllocCPUS,ReqMem,NTasks,NNodes
JobID         MaxVMSize  MaxVMSizeNode  MaxVMSizeNode  AveVMSize     MaxRSS MaxRSSNode MaxRSSTask     AveRSS  AllocCPUS     ReqMem   NTasks   NNodes 
------------ ---------- -------------- -------------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- -------- -------- 
273186_0                                                                                                              4        32G                 1 
273186_0.ba+  58504736K           x176           x176  58504736K  58362584K       x176          0  58362584K          4                   1        1 
273186_0.ex+        60K           x176           x176        60K        60K       x176          0        60K          4                   1        1