| Summary: | Free memory reported in sinfo not using available memory | ||
|---|---|---|---|
| Product: | Slurm | Reporter: | sysadmin |
| Component: | User Commands | Assignee: | Marcin Stolarek <mstolarek> |
| Status: | RESOLVED INFOGIVEN | QA Contact: | Ben Roberts <bproberts> |
| Severity: | 4 - Minor Issue | ||
| Priority: | --- | CC: | mstolarek |
| Version: | 21.08.3 | ||
| Hardware: | Linux | ||
| OS: | Linux | ||
| Site: | Allen Institute | Slinky Site: | --- |
| Alineos Sites: | --- | Atos/Eviden Sites: | --- |
| Confidential Site: | --- | Coreweave sites: | --- |
| Cray Sites: | --- | DS9 clusters: | --- |
| Google sites: | --- | HPCnow Sites: | --- |
| HPE Sites: | --- | IBM Sites: | --- |
| NOAA SIte: | --- | NoveTech Sites: | --- |
| Nvidia HWinf-CS Sites: | --- | OCF Sites: | --- |
| Recursion Pharma Sites: | --- | SFW Sites: | --- |
| SNIC sites: | --- | Tzag Elita Sites: | --- |
| Linux Distro: | --- | Machine Name: | |
| CLE Version: | Version Fixed: | ||
| Target Release: | --- | DevPrio: | --- |
| Emory-Cloud Sites: | --- | ||
Nathaniel, You are correct the number shown as FreeMem in sinfo and scontrol show node is the one you can check using free command. It's similar to CPULoad (the one from uptime command), we're gathering those numbers by slurmd (on registration and node ping RPC) and provide them for user/admin ease, but we don't use those in job scheduling. So jobs are allocated memory based on RealMemory parameter configured in slurm.conf, same applies to LLN (least loaded nodes) scheduling where we check the number of allocated CPUs not actually the load(from uptime output). cheers, Marcin Makes sense form what I was seeing- can I put a feature request in to change this to available memory? "free" memory is a fairly useless metric and we don't really have an option to tame the OS caching (doesn't really make sense to either...). Also figured the static "realmem" was being used for scheduling- but seeing "free" reported gave me some pause. Thanks for the response! If I don't hear back before the weekend- hope you have a good weekend! -Nathaniel Today the memory information in scontrol show node looks like: >RealMemory=15000 AllocMem=0 FreeMem=120 >MemSpecLimit=10 so we present a memory configured on the node (RealMemory), memory allocated to jobs (AllocMem), "free" memory as defined by OS (sysinfo.freeram[1]) caches and bufers impact on the definition is not specific to Slurm(FreeMem) and if configured MemSpecLimit[2]. >can I put a feature request in to change this to available memory? I don't think we can just replace the value, since it's there for many years and it will very likely break scripts used by Slurm users. The value of "available" memory isn't really stored in Slurm structures - it technically comes from the values already shown by `scontrol show node`. Same applies to the today state for CPUs and gres - we show total and allocated counters (and load for CPU, like FreeMem). Having all that in mind I'm not convinced that we should add another counter for memory. What do you think? cheers, Marcin [1]https://man7.org/linux/man-pages/man2/sysinfo.2.html [2]https://slurm.schedmd.com/slurm.conf.html#OPT_MemSpecLimit I don't think we can just replace the value, since it's there for many years and it will very likely break scripts used by Slurm users.
The value of "available" memory isn't really stored in Slurm structures - it technically comes from the values already shown by `scontrol show node`. Same applies to the today state for CPUs and gres - we show total and allocated counters (and load for CPU, like FreeMem). Having all that in mind I'm not convinced that we should add another counter for memory.
> I'd actually argue that it would fix quite a few scripts unless units change here... but they do not appear to. Using pestat as an example script that would be impacted, we see a presentation of the wrong metric for temporarily consumed vs. available memory (45G "free" vs 500G actually "available" for jobs). I could see available memory being presented as a quick freeram + bufferram equation.. as it does indeed look as if "available" is not included. Bets that "free -m" does something similar? :)
Note: If freemem is too slow to flush out as a job starts... I stand corrected, but slurm isn't using that for scheduling... so I am guessing that is not the case.
my $.02 here- I think this is a quick win for folks that use this number to gain a view of their cluster... and my eyes don't see why anyone would be wanting the "free" memory metric here. Disclaimer: I don't know every script, and understand caution in changing an old feature.
Created attachment 27098 [details] v1 >[...] Using pestat as an example script that would be impacted, I see the point, however, it looks like a bug in pestat to me. Per scontrol show node manual[1]: >FreeMem > The total memory, in MB, currently free on the node as reported by the OS. but sinfo has only: >FreeMem > Free memory of a node. I think we can improve sinfo docs to avoid the confusion, but the value was intentionally gathered by sysinfo when added by a contribution commit (a714b71c36) - so at least this site was using that as it is today[3]. From the long run perspective it's a breaking change. >Bets that "free -m" does something similar? :) I'd expect the number to be the same, on our side we use sysinfo routine (it's source vary between OSes). For free it may be also specific implementation dependent a quick look on one of the implementations I've notice they rely on /proc/meminfo interface. I'm sending the attached documentation patch to our QA queue. cheers, Marcin [1]https://slurm.schedmd.com/scontrol.html#OPT_FreeMem [2]https://slurm.schedmd.com/sinfo.html#OPT_FreeMem [3]https://github.com/SchedMD/slurm/commit/a714b71c366e94ebdc5235d7141819e4c7fdbd03 Nathaniel, We made some improvements to our docs: 75d47e9f6c5[1] and f91eb193ed94[2]. Those are going to be public in next minor release of Slurm - 22.05.5. Let me know if that makes it more clear. cheers, Marcin [1]https://github.com/SchedMD/slurm/commit/75d47e9f6c52ff629b53f298faa7786add2d971f [2]https://github.com/SchedMD/slurm/commit/f91eb193ed944de457ab8669a6bd19905422e293 Is there anything else I can help you with? In case of no reply I'll close the ticket as information given. cheers, Marcin |
Looking at sinfo's idea of free memory and seeing that it reports free memory... but I claim it should be reporting on available memory. How this pans out on my side: I am trialing a tool "pestat" which is using the output of sinfo to show hoe much memory each node has available. Example: Hostname Partition Node Num_CPU CPUload Memsize Freemem Joblist State Use/Tot (15min) (MB) (MB) JobID(JobArrayID) User ... n296 celltypes idle 0 112 0.07 515466 47414* This comes from parsing of: sinfo -h --node=n296 -O "NodeList:30,Partition:30,CPUsState:30,CPUsLoad:30,Memory:30,FreeMem:30,StateCompact:30,Threads:30,Gres:30" Looking at free -m: n296 ~]$ free -m total used free shared buff/cache available Mem: 515466 16700 47399 671 451366 496719 Swap: 4095 1760 2335 We see that FreeMem is coming from "free", when it should be using "available" The reason for "free" (in free -m) being inaccurate is that linux uses ram as a cache until it is needed elsewhere, so in this case... we actually have 496G of ram available... but slurm reports only 47G free. a better in-depth review of this is here: https://www.linuxatemyram.com/ Any chance we can get free to use available? Also- how does this play out in terms of scheduling? Is this just seen during sinfo queries? or is slurmctld using this metric to allocate jobs to nodes with "free" ram (I was under the impression this is just the static allocation... but double checking here)? Thanks for your time! -Nathaniel