Ticket 9523 - slurmd node configuration: wrong number of boards on AMD Rome
Summary: slurmd node configuration: wrong number of boards on AMD Rome
Status: RESOLVED FIXED
Alias: None
Product: Slurm
Classification: Unclassified
Component: slurmd (show other tickets)
Version: 20.02.3
Hardware: Linux Linux
: 4 - Minor Issue
Assignee: Marcin Stolarek
QA Contact:
URL:
Depends on:
Blocks:
 
Reported: 2020-08-05 18:28 MDT by Felix Abecassis
Modified: 2020-08-24 01:43 MDT (History)
2 users (show)

See Also:
Site: NVIDIA (PSLA)
Slinky Site: ---
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
Google sites: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
NoveTech Sites: ---
Nvidia HWinf-CS Sites: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Tzag Elita Sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed: 20.02.5 20.11.0pre1
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---

Attachments
v2 (1.34 KB, patch)
2020-08-10 13:19 MDT, Marcin Stolarek 		Details | Diff
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this ticket.
Description Felix Abecassis 2020-08-05 18:28:07 MDT 
On a AMD Rome (Zen 2 EPYC) system in mode NPS=4 (4 NUMA nodes per Socket),  "slurmd -C" reports 8 Boards and 2 SocketsPerBoard.

$ slurmd -C
NodeName=XXXXX CPUs=256 Boards=8 SocketsPerBoard=2 CoresPerSocket=64 ThreadsPerCore=2 RealMemory=2064082

This is not correct, the system has 1 Board, 2 sockets and thus a total of 8 NUMA nodes. The number of cores per socket is correct.

Looking at the output from "lstopo", the topology seems to be detected correctly, with a hierarchy of Package -> Group -> NUMANode -> Core. There are as many Groups as there are NUMANodes.
Comment 2 Marcin Stolarek 2020-08-10 06:43:34 MDT 
 Felix,

I'm looking into this. I think that you're correct - groups inside the package in terms of hwloc should not be interpreted as separate boards as done today.

I'll have to check the details of hwloc spec to figure out how to best address the case. Unfortunately, hwloc doesn't have a separate type for boards so we have to use groups here. 

Just to be sure - you can always override discovered node topology in slurm.conf adding SlurmdParameters=config_overrides.

cheers,
Marcin
Comment 5 Felix Abecassis 2020-08-10 10:10:55 MDT 
> Just to be sure - you can always override discovered node topology in slurm.conf adding SlurmdParameters=config_overrides.

Yes, it's more a low-priority FYI. It doesn't impact us right now.
Comment 6 Marcin Stolarek 2020-08-10 13:19:24 MDT 
Created attachment 15378 [details]
v2

Felix,

Could you please apply the attached patch a verify if the issue is fixed in the cases you were testing?

cheers,
Marcin
Comment 7 Felix Abecassis 2020-08-10 14:08:10 MDT 
Yes, your patch seems to work fine on this system, thanks!

NodeName=XXXXX CPUs=256 Boards=1 SocketsPerBoard=2 CoresPerSocket=64 ThreadsPerCore=2 RealMemory=2064085
Comment 11 Marcin Stolarek 2020-08-24 01:43:32 MDT 
Felix,

The fix got merged to our public repository[1] and will be released in 20.02.5.
Should you have any questions, please reopen the case.

cheers,
Marcin

[1]https://github.com/SchedMD/slurm/commit/6566c1b1c1735768fb4beff9566c9dd894ec44d0