Ticket 15077

Summary: Free memory reported in sinfo not using available memory
Product: Slurm Reporter: sysadmin
Component: User CommandsAssignee: Marcin Stolarek <mstolarek>
Status: RESOLVED INFOGIVEN QA Contact: Ben Roberts <bproberts>
Severity: 4 - Minor Issue    
Priority: --- CC: mstolarek
Version: 21.08.3   
Hardware: Linux   
OS: Linux   
Site: Allen Institute Slinky Site: ---
Alineos Sites: --- Atos/Eviden Sites: ---
Confidential Site: --- Coreweave sites: ---
Cray Sites: --- DS9 clusters: ---
Google sites: --- HPCnow Sites: ---
HPE Sites: --- IBM Sites: ---
NOAA SIte: --- NoveTech Sites: ---
Nvidia HWinf-CS Sites: --- OCF Sites: ---
Recursion Pharma Sites: --- SFW Sites: ---
SNIC sites: --- Tzag Elita Sites: ---
Linux Distro: --- Machine Name:
CLE Version: Version Fixed:
Target Release: --- DevPrio: ---
Emory-Cloud Sites: ---

Description sysadmin 2022-09-29 15:54:37 MDT
Looking at sinfo's idea of free memory and seeing that it reports free memory... but I claim it should be reporting on available memory.

How this pans out on my side:
I am trialing a tool "pestat" which is using the output of sinfo to show hoe much memory each node has available.

Example:
Hostname       Partition     Node Num_CPU  CPUload  Memsize  Freemem  Joblist
                            State Use/Tot  (15min)     (MB)     (MB)  JobID(JobArrayID) User ...
n296           celltypes    idle    0 112    0.07    515466    47414*

This comes from parsing of:
sinfo -h --node=n296 -O "NodeList:30,Partition:30,CPUsState:30,CPUsLoad:30,Memory:30,FreeMem:30,StateCompact:30,Threads:30,Gres:30"

Looking at free -m:
n296 ~]$ free -m
              total        used        free      shared  buff/cache   available
Mem:         515466       16700       47399         671      451366      496719
Swap:          4095        1760        2335

We see that FreeMem is coming from "free", when it should be using "available"

The reason for "free" (in free -m) being inaccurate is that linux uses ram as a cache until it is needed elsewhere, so in this case... we actually have 496G of ram available... but slurm reports only 47G free.

a better in-depth review of this is here: https://www.linuxatemyram.com/

Any chance we can get free to use available? Also- how does this play out in terms of scheduling? Is this just seen during sinfo queries? or is slurmctld using this metric to allocate jobs to nodes with "free" ram (I was under the impression this is just the static allocation... but double checking here)?

Thanks for your time!
-Nathaniel
Comment 1 Marcin Stolarek 2022-09-30 02:24:01 MDT
Nathaniel,

You are correct the number shown as FreeMem in sinfo and scontrol show node is the one you can check using free command. It's similar to CPULoad (the one from uptime command), we're gathering those numbers by slurmd (on registration and node ping RPC) and provide them for user/admin ease, but we don't use those in job scheduling.

So jobs are allocated memory based on RealMemory parameter configured in slurm.conf, same applies to LLN (least loaded nodes) scheduling where we check the number of allocated CPUs not actually the load(from uptime output).

cheers,
Marcin
Comment 2 sysadmin 2022-09-30 09:52:34 MDT
Makes sense form what I was seeing- can I put a feature request in to change this to available memory? "free" memory is a fairly useless metric and we don't really have an option to tame the OS caching (doesn't really make sense to either...). 

Also figured the static "realmem" was being used for scheduling- but seeing "free" reported gave me some pause.

Thanks for the response! If I don't hear back before the weekend- hope you have a  good weekend!
-Nathaniel
Comment 3 Marcin Stolarek 2022-10-03 01:33:30 MDT
Today the memory information in scontrol show node looks like:
>RealMemory=15000 AllocMem=0 FreeMem=120
>MemSpecLimit=10
so we present a memory configured on the node (RealMemory), memory allocated to jobs (AllocMem), "free" memory as defined by OS (sysinfo.freeram[1]) caches and bufers impact on the definition is not specific to Slurm(FreeMem) and if configured MemSpecLimit[2].

>can I put a feature request in to change this to available memory?
I don't think we can just replace the value, since it's there for many years and it will very likely break scripts used by Slurm users.
The value of "available" memory isn't really stored in Slurm structures - it technically comes from the values already shown by `scontrol show node`. Same applies to the today state for CPUs and gres - we show total and allocated counters (and load for CPU, like FreeMem). Having all that in mind I'm not convinced that we should add another counter for memory.

What do you think?

cheers,
Marcin
[1]https://man7.org/linux/man-pages/man2/sysinfo.2.html
[2]https://slurm.schedmd.com/slurm.conf.html#OPT_MemSpecLimit
Comment 4 sysadmin 2022-10-03 12:13:36 MDT
I don't think we can just replace the value, since it's there for many years and it will very likely break scripts used by Slurm users.
The value of "available" memory isn't really stored in Slurm structures - it technically comes from the values already shown by `scontrol show node`. Same applies to the today state for CPUs and gres - we show total and allocated counters (and load for CPU, like FreeMem). Having all that in mind I'm not convinced that we should add another counter for memory.

> I'd actually argue that it would fix quite a few scripts unless units change here... but they do not appear to. Using pestat as an example script that would be impacted, we see a presentation of the wrong metric for temporarily consumed vs. available memory (45G "free" vs 500G actually "available" for jobs). I could see available memory being presented as a quick freeram + bufferram equation.. as it does indeed look as if "available" is not included. Bets that "free -m" does something similar? :) 

Note: If freemem is too slow to flush out as a job starts... I stand corrected, but slurm isn't using that for scheduling... so I am guessing that is not the case.

my $.02 here- I think this is a quick win for folks that use this number to gain a view of their cluster... and my eyes don't see why anyone would be wanting the "free" memory metric here. Disclaimer: I don't know every script, and understand caution in changing an old feature.
Comment 5 Marcin Stolarek 2022-10-04 00:30:12 MDT
Created attachment 27098 [details]
v1

>[...] Using pestat as an example script that would be impacted,

I see the point, however, it looks like a bug in pestat to me. Per scontrol show node manual[1]:
>FreeMem
>      The total memory, in MB, currently free on the node as reported by the OS.
but sinfo has only:
>FreeMem
>      Free memory of a node.
I think we can improve sinfo docs to avoid the confusion, but the value was intentionally gathered by sysinfo when added by a contribution commit (a714b71c36) - so at least this site was using that as it is today[3]. From the long run perspective it's a breaking change.

>Bets that "free -m" does something similar? :) 
I'd expect the number to be the same, on our side we use sysinfo routine (it's source vary between OSes). For free it may be also specific implementation dependent a quick look on one of the implementations I've notice they rely on /proc/meminfo interface.

I'm sending the attached documentation patch to our QA queue.

cheers,
Marcin

[1]https://slurm.schedmd.com/scontrol.html#OPT_FreeMem
[2]https://slurm.schedmd.com/sinfo.html#OPT_FreeMem
[3]https://github.com/SchedMD/slurm/commit/a714b71c366e94ebdc5235d7141819e4c7fdbd03
Comment 7 Marcin Stolarek 2022-10-04 19:16:57 MDT
Nathaniel,

We made some improvements to our docs: 75d47e9f6c5[1] and f91eb193ed94[2]. Those are going to be public in next minor release of Slurm - 22.05.5.

Let me know if that makes it more clear.

cheers,
Marcin
[1]https://github.com/SchedMD/slurm/commit/75d47e9f6c52ff629b53f298faa7786add2d971f
[2]https://github.com/SchedMD/slurm/commit/f91eb193ed944de457ab8669a6bd19905422e293
Comment 8 Marcin Stolarek 2022-11-09 12:09:42 MST
Is there anything else I can help you with?

In case of no reply I'll close the ticket as information given.

cheers,
Marcin