Ticket 13401

Summary:	Jobs submitted against a future reservation are not being accepted
Product:	Slurm	Reporter:	Mallick Arigapudi <mallick.arigapudi>
Component:	reservations	Assignee:	Scott Hilton <scott>
Status:	RESOLVED TIMEDOUT	QA Contact:
Severity:	3 - Medium Impact
Priority:	---
Version:	21.08.4
Hardware:	Linux
OS:	Linux
Site:	Intel CRT	Alineos Sites:	---
Atos/Eviden Sites:	---	Confidential Site:	---
Coreweave sites:	---	Cray Sites:	---
DS9 clusters:	---	HPCnow Sites:	---
HPE Sites:	---	IBM Sites:	---
NOAA SIte:	---	OCF Sites:	---
Recursion Pharma Sites:	---	SFW Sites:	---
SNIC sites:	---	Linux Distro:	---
Machine Name:		CLE Version:
Version Fixed:		Target Release:	---
DevPrio:	---	Emory-Cloud Sites:	---

Description Mallick Arigapudi 2022-02-10 10:05:10 MST

We have a reservation set to start on StartTime=2022-02-13T09:00:00

ReservationName=24096 StartTime=2022-02-13T09:00:00 EndTime=2022-02-14T09:00:00 Duration=1-00:00:00
   Nodes=eia[005-013,016,020-042,045,047-053..] NodeCnt=311 CoreCnt=22392 Features=(null) PartitionName=itarq Flags=FLEX,SPEC_NODES,MAGNETIC
   TRES=cpu=22392
   Users=xx,yy,zz Groups=(null) Accounts=(null) Licenses=(null) State=INACTIVE BurstBuffer=(null) Watts=n/a
   MaxStartDelay=(null)

Jobs submitted today, are not being accepted

$ bsub -P O -t 60 -N 147 -p itarq -C icx8360Y -l EIST=2401000 --reservation=24096 -Is sh
interactive
Project Area: D
Project Desc: DoD
srun --pty --exclusive   "-t" "60" "-N" "147" "-p" "itarq" "-C" "icx8360Y" "--reservation=24096" "sh"
srun: environment addon enabled
srun: found Resource_List_CRTDC_PROJECTCODE=D:DoD
srun: found Resource_List_EIST=2401000
srun: error: Unable to allocate resources: Requested node configuration is not available

If I try the same with currently available IDLE nodes, the job is accepted

$ bsub -P D:DoD -t 60 -N 7 -p itarq -C icx8360Y -l EIST=2401000 --reservation=24096 -Is sh
interactive
Project Area: D
Project Desc: DoD
srun --pty --exclusive   "-t" "60" "-N" "7" "-p" "itarq" "-C" "icx8360Y" "--reservation=24096" "sh"
srun: environment addon enabled
srun: found Resource_List_CRTDC_PROJECTCODE=D:DoD
srun: found Resource_List_EIST=2401000
srun: job 331728 queued and waiting for resources

The expectation is, the user should be able to submit a job against a reservation and expect the job get scheduled at the start time of the reservation, but not reject it out right.

Comment 1 Scott Hilton 2022-02-10 11:30:23 MST

Can I get your slurm.conf?

-Scott

Comment 2 Mallick Arigapudi 2022-02-10 11:37:03 MST

# cat /etc/slurm.conf
# slurm.conf file generated by configurator.html.
# Put this file on all nodes of your cluster.
# See the slurm.conf man page for more information.
#
SlurmctldHost=eslurm1
#
#DisableRootJobs=NO
EnforcePartLimits=YES
Epilog=/opt/slurm/current/etc/prolog.d/wrapper.XXXlog.sh
#EpilogSlurmctld=
#FirstJobId=1
#MaxJobId=999999
#GresTypes=
GroupUpdateForce=1
#GroupUpdateTime=600
JobFileAppend=1
#JobRequeue=1
#JobSubmitPlugins=lua
#KillOnBadExit=0
#LaunchType=launch/slurm
#Licenses=foo*4,bar
Licenses=fluent:6,daos:10
MailProg=/bin/false
#MaxJobCount=5000
#MaxStepCount=40000
#MaxTasksPerNode=128
MpiDefault=none
#MpiParams=ports=#-#
#PluginDir=/opt/slurm/current/etc/plugins
PlugStackConfig=/opt/slurm/current/etc/plugstack.conf
#PrivateData=jobs
ProctrackType=proctrack/linuxproc
Prolog=/opt/slurm/current/etc/prolog.d/wrapper.XXXlog.sh
#PrologFlags=Alloc
PrologFlags=Alloc,serial
PrologSlurmctld=/opt/slurm/crtdc/slurmctld.XXXlog.sh
#PropagatePrioProcess=0
PropagateResourceLimits=NONE
#PropagateResourceLimitsExcept=
RebootProgram=/bin/true
ReturnToService=0
#ReturnToService=2
#SallocDefaultCommand=
SlurmctldPidFile=/var/run/slurmctld.pid
SlurmctldPort=6817
SlurmdPidFile=/var/run/slurmd.pid
SlurmdPort=6818
SlurmdSpoolDir=/var/spool/slurmd
SlurmUser=slurm
#SlurmdUser=root
#SrunEpilog=/opt/slurm/current/bin/XXXlogue.py
#SrunProlog=/opt/slurm/current/bin/XXXlogue.py
StateSaveLocation=/opt/slurm/current/var/spool
SwitchType=switch/none
#TaskEpilog=/opt/slurm/current/bin/XXXlogue.py
#TaskPlugin=task/affinity
TaskPlugin=task/none
TaskPluginParam=none
#TaskProlog=/opt/slurm/current/bin/XXXlogue.py
#TopologyPlugin=topology/tree
#TmpFS=/tmp
#TrackWCKey=no
#TreeWidth=
#UnkillableStepProgram=
#UsePAM=0
#
#
# TIMERS
BatchStartTimeout=3000
CompleteWait=5
EpilogMsgTime=3000
PrologEpilogTimeout=3000
#GetEnvTimeout=2
#HealthCheckInterval=0
#HealthCheckProgram=
InactiveLimit=0
KillWait=30
#MessageTimeout=10
#ResvOverRun=0
MinJobAge=300
#OverTimeLimit=0
SlurmctldTimeout=120
SlurmdTimeout=300
#UnkillableStepTimeout=60
#VSizeFactor=0
Waittime=0
#
#
# SCHEDULING
#DefMemPerCPU=0
#MaxMemPerCPU=0
#SchedulerTimeSlice=30
SchedulerType=sched/backfill
SelectType=select/linear
#SelectType=select/cons_res
#SelectTypeParameters=CR_CORE
SchedulerParameters=nohold_on_prolog_fail,defer
#SelectTypeParameters=
#
#
# JOB PRIORITY
#PriorityFlags=
#PriorityType=priority/basic
# Added the below setting to favor high node count jobs first - MA Dec20 2021
#PriorityType=priority/multifactor - commented out today Jan 26th 2021
#PriorityDecayHalfLife=
#PriorityCalcPeriod=
#PriorityFavorSmall=
# Added the below setting to favor high node count jobs first - MA Dec20 2021
#PriorityFavorSmall=No - commented out today Jan 26th 2021
#PriorityMaxAge=
#PriorityUsageResetPeriod=
#PriorityWeightAge=
#PriorityWeightFairshare=
#PriorityWeightJobSize=
#PriorityWeightPartition=
#PriorityWeightQOS=
#
#
# LOGGING AND ACCOUNTING #
AccountingStorageEnforce=limits,qos
#AccountingStorageHost=
#AccountingStorageLoc=/opt/slurm/logs/slurm.log
#AccountingStoragePass=
#AccountingStoragePort=
AccountingStorageType=accounting_storage/slurmdbd
AccountingStorageUser=slurm
# AccountingStoreJobComment removed; changed to AccountingStoreFlags per remark on slurmctld startup
#AccountingStoreJobComment=YES
AccountingStoreFlags=job_comment
ClusterName=endeavour
#DebugFlags=
JobCompHost=localhost
#JobCompLoc=/opt/slurm/current/logs/slurm/jobs
#JobCompPass=
#JobCompPort=
JobCompType=jobcomp/mysql
#JobCompUser=
#JobContainerType=job_container/none
JobAcctGatherFrequency=task=0,energy=0,network=0,filesystem=0
JobAcctGatherType=jobacct_gather/linux
SlurmctldDebug=info
SlurmctldLogFile=/opt/slurm/current/logs/slurm/slurmctl.log
SlurmdDebug=info
SlurmdLogFile=/opt/slurm/current/logs/slurm/slurmd.log
SlurmSchedLogFile=/opt/slurm/current/logs/slurm/sched.log
SlurmSchedLogLevel=1
#
#
# POWER SAVE SUPPORT FOR IDLE NODES (optional)
#SuspendProgram=
#ResumeProgram=
#SuspendTimeout=
ResumeTimeout=1800
#ResumeRate=
#SuspendExcNodes=
#SuspendExcParts=
#SuspendRate=
#SuspendTime=
#
AuthInfo=cred_expire=240
#
# COMPUTE NODES

# DOWN NODES

# NODE LIST

NodeName=eca[001-012]  Boards=1 SocketsPerBoard=2 CoresPerSocket=48 State=UNKNOWN Feature=clxap9242,leaf5c,reconfig,corenode
NodeName=eca[013-014]  Boards=1 SocketsPerBoard=2 CoresPerSocket=48 State=UNKNOWN Feature=clxap9242,leaf5b,reconfig,corenode
NodeName=eca[015-018]  Boards=1 SocketsPerBoard=2 CoresPerSocket=48 State=UNKNOWN Feature=clxap9242,leaf5b,reconfig,corenode
NodeName=eca[019-024]  Boards=1 SocketsPerBoard=2 CoresPerSocket=48 State=UNKNOWN Feature=clxap9242,leaf5d,reconfig,corenode
NodeName=eca[025-028]  Boards=1 SocketsPerBoard=2 CoresPerSocket=48 State=UNKNOWN Feature=clxap9242,leaf5a,reconfig,corenode
NodeName=eca[029-030]  Boards=1 SocketsPerBoard=2 CoresPerSocket=48 State=UNKNOWN Feature=clxap9242,leaf5a,reconfig,corenode
NodeName=eca[031-033]  Boards=1 SocketsPerBoard=2 CoresPerSocket=48 State=UNKNOWN Feature=clxap9242,leaf5c,reconfig,corenode
NodeName=eca034  Boards=1 SocketsPerBoard=2 CoresPerSocket=48 State=FUTURE Feature=clxap9242,leaf5c,reconfig,corenode
NodeName=eca035        Boards=1 SocketsPerBoard=2 CoresPerSocket=48 State=FUTURE Feature=clxap9242,leaf5c,reconfig,corenode
NodeName=eca[038-043]  Boards=1 SocketsPerBoard=2 CoresPerSocket=48 State=UNKNOWN Feature=clxap9242,leaf5b,reconfig,corenode
NodeName=eca044        Boards=1 SocketsPerBoard=2 CoresPerSocket=48 State=FUTURE Feature=clxap9242,leaf5d,reconfig,corenode
NodeName=eca[046-048]  Boards=1 SocketsPerBoard=2 CoresPerSocket=48 State=UNKNOWN Feature=clxap9242,leaf5d,reconfig,corenode
NodeName=eca[049-054]  Boards=1 SocketsPerBoard=2 CoresPerSocket=48 State=UNKNOWN Feature=clxap9242,leaf5a,reconfig,corenode
NodeName=eca[055-056]  Boards=1 SocketsPerBoard=2 CoresPerSocket=48 State=UNKNOWN Feature=clxap9242,leaf5c,reconfig,corenode
NodeName=eca[073-079]  Boards=1 SocketsPerBoard=2 CoresPerSocket=32 State=UNKNOWN Feature=clxap9221,leaf13,reconfig,corenode
NodeName=eca080        Boards=1 SocketsPerBoard=2 CoresPerSocket=32 State=UNKNOWN Feature=clxap9221,leaf13,reconfig,corenode

NodeName=ecx[001-004]  Boards=1 SocketsPerBoard=4 CoresPerSocket=28 State=UNKNOWN Feature=cpx8388HL
NodeName=ecx[005]      Boards=1 SocketsPerBoard=4 CoresPerSocket=28 State=FUTURE  Feature=cpx8388HL
NodeName=ecx[007,009]  Boards=1 SocketsPerBoard=4 CoresPerSocket=28 State=FUTURE  Feature=cpx8388HL
NodeName=ecx008        Boards=1 SocketsPerBoard=4 CoresPerSocket=28 State=UNKNOWN Feature=cpx8388HL

NodeName=ega[001-002]  Boards=1 SocketsPerBoard=2 CoresPerSocket=24 State=UNKNOWN Feature=clx6252N,leaf11U,reconfig
NodeName=ega003        Boards=1 SocketsPerBoard=2 CoresPerSocket=24 State=FUTURE  Feature=clx6252N,leaf11U,reconfig
NodeName=ega[004-016]  Boards=1 SocketsPerBoard=2 CoresPerSocket=24 State=UNKNOWN Feature=clx6252N,leaf11U,reconfig
NodeName=ega018  Boards=1 SocketsPerBoard=2 CoresPerSocket=24 State=FUTURE Feature=clx6252N,leaf11U,reconfig
NodeName=ega[019-020]  Boards=1 SocketsPerBoard=2 CoresPerSocket=24 State=UNKNOWN Feature=clx6252N,leaf11U,reconfig
NodeName=ega[021-031]  Boards=1 SocketsPerBoard=2 CoresPerSocket=24 State=UNKNOWN Feature=clx6252N,leaf11L,reconfig
NodeName=ega[032-040]  Boards=1 SocketsPerBoard=2 CoresPerSocket=24 State=UNKNOWN Feature=clx6252N,leaf11L,reconfig
NodeName=ega[041-060]  Boards=1 SocketsPerBoard=2 CoresPerSocket=24 State=UNKNOWN Feature=clx6252N,leaf12U,reconfig
NodeName=ega[061-080]  Boards=1 SocketsPerBoard=2 CoresPerSocket=24 State=UNKNOWN Feature=clx6252N,leaf12L,reconfig

NodeName=ehx01         Boards=1 SocketsPerBoard=4 CoresPerSocket=18 State=UNKNOWN Feature=hsw4s8890,reconfig
NodeName=ehx02         Boards=1 SocketsPerBoard=4 CoresPerSocket=18 State=UNKNOWN Feature=hsw4s8890,reconfig
NodeName=ehx04         Boards=1 SocketsPerBoard=4 CoresPerSocket=18 State=UNKNOWN Feature=hsw4s8890,reconfig

NodeName=eea[001-002]  Boards=1 SocketsPerBoard=2 CoresPerSocket=56 State=FUTURE  Feature=spr56cpq,sleaf107,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eea003        Boards=1 SocketsPerBoard=2 CoresPerSocket=56 State=FUTURE  Feature=spr56cpq,sleaf107,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eea[004-005]  Boards=1 SocketsPerBoard=2 CoresPerSocket=56 State=FUTURE  Feature=spr56cpq,sleaf107,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eea006        Boards=1 SocketsPerBoard=2 CoresPerSocket=56 State=FUTURE Feature=spr56cpq,sleaf107,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eea007        Boards=1 SocketsPerBoard=2 CoresPerSocket=56 State=FUTURE  Feature=spr56cpq,sleaf107,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eea008        Boards=1 SocketsPerBoard=2 CoresPerSocket=56 State=FUTURE Feature=spr56cpq,sleaf107,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eea009        Boards=1 SocketsPerBoard=2 CoresPerSocket=56 State=FUTURE  Feature=spr56cpq,sleaf107,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eea[010-018]  Boards=1 SocketsPerBoard=2 CoresPerSocket=56 State=FUTURE Feature=spr56cpq,sleaf107,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eea019        Boards=1 SocketsPerBoard=2 CoresPerSocket=56 State=FUTURE  Feature=spr56cpq,sleaf107,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eea[020-022]  Boards=1 SocketsPerBoard=2 CoresPerSocket=56 State=FUTURE Feature=spr56cpq,sleaf107,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eea[023-024]  Boards=1 SocketsPerBoard=2 CoresPerSocket=56 State=FUTURE Feature=spr56cpq,sleaf107,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eea[025-026]  Boards=1 SocketsPerBoard=2 CoresPerSocket=56 State=FUTURE  Feature=spr56cpq,sleaf104,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eea[027-028]  Boards=1 SocketsPerBoard=2 CoresPerSocket=56 State=FUTURE Feature=spr56cpq,sleaf107,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
# eea029 deepdive does not boot
NodeName=eea[029-030]  Boards=1 SocketsPerBoard=2 CoresPerSocket=56 State=FUTURE  Feature=spr56cpq,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eea[031-034]  Boards=1 SocketsPerBoard=2 CoresPerSocket=56 State=FUTURE Feature=spr56cpqfat,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eea[035-036]  Boards=1 SocketsPerBoard=2 CoresPerSocket=56 State=FUTURE Feature=spr56cpqfat,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eea037        Boards=1 SocketsPerBoard=2 CoresPerSocket=56 State=FUTURE  Feature=spr56cpqfat,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eea[038-040]  Boards=1 SocketsPerBoard=2 CoresPerSocket=56 State=FUTURE Feature=spr56cpqfat,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8

NodeName=eia001        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=FUTURE  Feature=icx8360Y,reconfig,leaf109,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii002        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,icx8360Yfx,reconfig,leaf109,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia[003-004]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=FUTURE  Feature=icx8360Y,reconfig,leaf109,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia[005-013]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf109,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii[014-015]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,icx8360Yfx,reconfig,leaf109,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia016        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf109,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii017        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf109,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii018        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,icx8360Yfx,reconfig,leaf109,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii019        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,icx8360Yfx,reconfig,leaf109,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8

NodeName=eia[020-023]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf110,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia[024-026]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf110,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia[027-029]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf110,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia[030-036]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf110,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8

NodeName=eia[037-042]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf111,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii[043-044]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf111,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia045        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf111,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii046        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf111,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia[047-049]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf111,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia[050-053]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf111,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii[050-053]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf111,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii054        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf111,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8

NodeName=eia[055-056]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf112,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii057        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,icx8360Yfx,reconfig,leaf112,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii058        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=FUTURE  Feature=icx8360Y,reconfig,leaf112,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii059        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,icx8360Yfx,reconfig,leaf112,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia[060-068]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf112,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii069        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf112,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia[070-071]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf112,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii072        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,icx8360Yfx,reconfig,leaf112,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8

NodeName=eia073        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf113,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii074        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,icx8360Yfx,reconfig,leaf113,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia[075-077]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf113,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia078  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf113,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia079  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=FUTURE Feature=icx8360Y,reconfig,leaf113,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia[080-081]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf113,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia[082-083]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf113,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia[084-087]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf113,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii088        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,icx8360Yfx,reconfig,leaf113,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia[089-093]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf113,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8

NodeName=eia[094-096]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf114,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii097        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf114,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia[098,100]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf114,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia101  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=FUTURE Feature=icx8360Y,reconfig,leaf114,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia[102-104]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf114,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia105        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=FUTURE  Feature=icx8360Y,reconfig,leaf114,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia[106-107]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf114,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia108        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=FUTURE  Feature=icx8360Y,reconfig,leaf114,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8

NodeName=eia[109-111]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf115,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia[112-114]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf115,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii115        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,icx8360Yfx,reconfig,leaf115,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia116        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf115,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia117        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=FUTURE  Feature=icx8360Y,reconfig,leaf115,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
#eia117 rma BMC failure
NodeName=eia[118-123]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf115,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia124  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=FUTURE Feature=icx8360Y,reconfig,leaf115,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia[125-126]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf116,icx8360Yf3,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii[127-137]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf116,icx8360Yf3,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii[138-142]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf116,icx8360Yf3,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii143        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=FUTURE  Feature=icx8360Y,reconfig,leaf116,icx8360Yf3,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii144        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf116,icx8360Yf3,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8

NodeName=eia[145-146]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf209,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii147        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,icx8360Yfx,reconfig,leaf209,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia148        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=FUTURE  Feature=icx8360Y,reconfig,leaf209,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia[149-154]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf209,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia[155-162]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf209,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8

NodeName=eia[163-168]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf210,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia[169-173]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf210,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia[174-178]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf210,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii179        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,icx8360Yfx,reconfig,leaf210,x,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia180        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf210,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8

NodeName=eia[181-193]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf211,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia[194-197]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf211,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia[198-198]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf211,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8

NodeName=eia[199-200]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf212,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia[201-202]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf212,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia[204-211]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf212,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii212        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf212,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia213        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf212,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii214        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,icx8360Yfx,reconfig,leaf212,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia[215-216]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf212,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8

NodeName=eii[217-226]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf213,icx8360Yf3,icx8360Yopa,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii[227-230]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf213,icx8360Yf3,icx8360Yopa,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii[231-234]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf213,icx8360Yf3,icx8360Yopa,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii235        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf214,icx8360Yf3,icx8360Yopa,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
# NotInTheRack eii236
NodeName=eii236        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=FUTURE  Feature=icx8360Y,reconfig,leaf214,icx8360Yf3,icx8360Yopa,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii[237-248]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf214,icx8360Yf3,icx8360Yopa,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii[249-252]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf214,icx8360Yf3,icx8360Yopa,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8

NodeName=eii[253-258]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf215,icx8360Yf3,icx8360Yopa,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii[259-260]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf215,icx8360Yf3,icx8360Yopa,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii261        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf215,icx8360Yf3,icx8360Yopa,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii262        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf215,icx8360Yf3,icx8360Yopa,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii263        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf215,icx8360Yf3,icx8360Yopa,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii[264-270]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf215,icx8360Yf3,icx8360Yopa,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii[271-272]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf216,icx8360Yf3,icx8360Yopa,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
# Failed during Nemirov testing 11/17/21
NodeName=eii[273-277]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf216,icx8360Yf3,icx8360Yopa,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii278        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=FUTURE Feature=icx8360Y,reconfig,leaf216,icx8360Yf3,icx8360Yopa,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii[279-286]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf216,icx8360Yf3,icx8360Yopa,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia287        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf216,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia288        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf216,icx8360Yf2,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8

NodeName=eit289        Boards=1 SocketsPerBoard=2 CoresPerSocket=32 State=UNKNOWN Feature=icx8352Y,reconfig,leaf310a,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eit290        Boards=1 SocketsPerBoard=2 CoresPerSocket=32 State=UNKNOWN Feature=icx8352Y,reconfig,leaf310a,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eit291        Boards=1 SocketsPerBoard=2 CoresPerSocket=32 State=FUTURE  Feature=icx8352Y,reconfig,leaf310a,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eit292        Boards=1 SocketsPerBoard=2 CoresPerSocket=32 State=UNKNOWN Feature=icx8352Y,reconfig,leaf310a,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eit293        Boards=1 SocketsPerBoard=2 CoresPerSocket=32 State=UNKNOWN Feature=icx8352Y,reconfig,leaf310a,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eit294        Boards=1 SocketsPerBoard=2 CoresPerSocket=32 State=UNKNOWN Feature=icx8352Y,reconfig,leaf310a,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eit[295-297]  Boards=1 SocketsPerBoard=2 CoresPerSocket=32 State=UNKNOWN Feature=icx8352Y,reconfig,leaf310a,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eit[298-300]  Boards=1 SocketsPerBoard=2 CoresPerSocket=32 State=UNKNOWN Feature=icx8352Y,reconfig,leaf310a,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eit[301-304]  Boards=1 SocketsPerBoard=2 CoresPerSocket=32 State=UNKNOWN Feature=icx8352M,reconfig,leaf310b,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eit305        Boards=1 SocketsPerBoard=2 CoresPerSocket=32 State=FUTURE  Feature=icx8352Y,reconfig,leaf310b,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eit306        Boards=1 SocketsPerBoard=2 CoresPerSocket=32 State=UNKNOWN Feature=icx8352M,reconfig,leaf310b,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8

NodeName=eii307  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=FUTURE Feature=icx8360Y,reconfig,leaf311,icx8360Yf3,icx8360YatsB0,icx8360Yats2xB0,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii[308-314]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf311,icx8360Yf3,icx8360YatsB0,icx8360Yats2xB0,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
# NotInTheRack  eii315
NodeName=eii315        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=FUTURE  Feature=icx8360Yats2xB0,icx8360Y,reconfig,leaf311,icx8360Yf3,icx8360YatsB0,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii[316-321]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Yats2xB0,icx8360Y,reconfig,leaf311,icx8360Yf3,icx8360YatsB0,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii322        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=FUTURE  Feature=icx8360Yats2xB0,icx8360Y,reconfig,leaf311,icx8360Yf3,icx8360YatsB0,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii[323-323]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Yats2xB0,icx8360Y,reconfig,leaf311,icx8360Yf3,icx8360YatsB0,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii324        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,icx8360Yats2xB0,icx8360Ytest,reconfig,leaf311,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8

NodeName=eii[325-328]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,icx8360Yats2xB0,icx8360Ytest,reconfig,leaf311,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii[329-330]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,icx8360Yats2xB0,icx8360Ytest,reconfig,leaf311,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii[331-334]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,icx8360YatsB0,icx8360Ytest,reconfig,leaf311,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii[335-337]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,icx8360YatsB0,icx8360Ytest,reconfig,leaf311,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii338        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,icx8360YatsB0,icx8360Ytest,reconfig,leaf311,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii339        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,icx8360YatsB0,icx8360Ytest,reconfig,leaf311,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii[340-341]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,icx8360YatsB0,icx8360Ytest,reconfig,leaf311,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii[342-343]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,icx8360YatsB0,icx8360Ytest,reconfig,leaf311,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eii[344-354]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360YatsB0,icx8360Y,reconfig,leaf313,icx8360Yf3,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8

NodeName=eid[356-357]  Boards=1 SocketsPerBoard=2 CoresPerSocket=32 State=UNKNOWN Feature=icx6338,reconfig,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eig358        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8352V,reconfig,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eig[359,360]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=FUTURE  Feature=icx8352V,reconfig,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eie361        Boards=1 SocketsPerBoard=2 CoresPerSocket=40 State=FUTURE  Feature=icx8380,reconfig,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eie[362-369]  Boards=1 SocketsPerBoard=2 CoresPerSocket=40 State=UNKNOWN Feature=icx8380,reconfig,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eif370        Boards=1 SocketsPerBoard=2 CoresPerSocket=32 State=FUTURE  Feature=icx8358,reconfig,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eif371        Boards=1 SocketsPerBoard=2 CoresPerSocket=32 State=FUTURE  Feature=icx8358,reconfig,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
# deep dive performance
NodeName=eif[372-375]  Boards=1 SocketsPerBoard=2 CoresPerSocket=32 State=UNKNOWN Feature=icx8358,reconfig,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eif376        Boards=1 SocketsPerBoard=2 CoresPerSocket=32 State=FUTURE  Feature=icx8358,reconfig,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eif[377-378]  Boards=1 SocketsPerBoard=2 CoresPerSocket=32 State=UNKNOWN Feature=icx8358,reconfig,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eil[390-393]  Boards=1 SocketsPerBoard=2 CoresPerSocket=16 State=UNKNOWN Feature=icx6354,reconfig,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eif[394-396]  Boards=1 SocketsPerBoard=2 CoresPerSocket=32 State=UNKNOWN Feature=icx8358,reconfig,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8

NodeName=eii379        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf316,icx8360Yf3,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
# Failed during Nemirov testing 11/17/21
NodeName=eii[380-383]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf316,icx8360Yf3,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8

NodeName=eik[386-389]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx6342,reconfig,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia[501-502]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=FUTURE  Feature=icx8360Y,reconfig,leaf500,icx8360Yf2,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia[503-505]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf500,icx8360Yf2,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia[506-510]  Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Y,reconfig,leaf500,icx8360Yf2,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8

NodeName=eim511        Boards=1 SocketsPerBoard=2 CoresPerSocket=16 State=UNKNOWN Feature=icx6346ats2xB0,reconfig,icx6346,icx6346atsB0,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eim512        Boards=1 SocketsPerBoard=2 CoresPerSocket=16 State=UNKNOWN Feature=icx6346ats2xB0,reconfig,icx6346,icx6346atsB0,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eim513        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=FUTURE  Feature=icx6346ats2xB0,reconfig,icx6346,icx6346atsB0,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eim[514-517]  Boards=1 SocketsPerBoard=2 CoresPerSocket=16 State=UNKNOWN Feature=icx6346ats2xB0,reconfig,icx6346,icx6346atsB0,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eim[518-520]  Boards=1 SocketsPerBoard=2 CoresPerSocket=16 State=UNKNOWN Feature=icx6346ats2xB0,reconfig,icx6346,icx6346atsB0,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eim[521-523]  Boards=1 SocketsPerBoard=2 CoresPerSocket=16 State=FUTURE  Feature=icx6346ats2xB0,reconfig,icx6346,icx6346atsB0,icx6346fat,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eim524        Boards=1 SocketsPerBoard=2 CoresPerSocket=16 State=UNKNOWN Feature=icx6346ats2xB0,reconfig,icx6346,icx6346atsB0,icx6346fat,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eim525        Boards=1 SocketsPerBoard=2 CoresPerSocket=16 State=FUTURE  Feature=icx6346ats2xB0,reconfig,icx6346,icx6346atsB0,icx6346fat,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eim526        Boards=1 SocketsPerBoard=2 CoresPerSocket=16 State=UNKNOWN Feature=icx6346ats2xB0,reconfig,icx6346,icx6346atsB0,icx6346fat,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=einv[527-528] Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=FUTURE  Feature=icx8360YA80GB,icx8360Y,reconfig,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eia529        Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360YatsB0,icx8360Y,reconfig,leaf324,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8

NodeName=einv[001-003] Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Ya100,reconfig,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=einv004       Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=FUTURE  Feature=icx8360Ya100,reconfig,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=einv[005-010] Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Ya100,reconfig,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=einv[011-012] Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Ya100,reconfig,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=einv[013-017] Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=UNKNOWN Feature=icx8360Ya100,reconfig,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=einv018       Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=FUTURE  Feature=icx8360Ya100,reconfig,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8

NodeName=ekln[01-02]   Boards=1 SocketsPerBoard=2 CoresPerSocket=12 State=FUTURE  Feature=rebootctrl
NodeName=elb[01-03]    Boards=1 SocketsPerBoard=2 CoresPerSocket=18 State=UNKNOWN Feature=bdw2697,reconfig
NodeName=elb04         Boards=1 SocketsPerBoard=2 CoresPerSocket=22 State=UNKNOWN Feature=bdw2699,reconfig
NodeName=elbx[02,03]   Boards=1 SocketsPerBoard=4 CoresPerSocket=24 State=UNKNOWN Feature=bdw4s8890,reconfig
NodeName=elc02         Boards=1 SocketsPerBoard=2 CoresPerSocket=28 State=UNKNOWN Feature=mtg8280L,reconfig
NodeName=elc03         Boards=1 SocketsPerBoard=2 CoresPerSocket=20 State=UNKNOWN Feature=mtg6248,reconfig
NodeName=elc04         Boards=1 SocketsPerBoard=2 CoresPerSocket=24 State=UNKNOWN Feature=mtg8260L,reconfig
NodeName=elh[01-03]    Boards=1 SocketsPerBoard=2 CoresPerSocket=14 State=UNKNOWN Feature=hsw,hsw2697,reconfig
NodeName=elh04         Boards=1 SocketsPerBoard=2 CoresPerSocket=18 State=UNKNOWN Feature=hsw2699,reconfig
NodeName=els02         Boards=1 SocketsPerBoard=2 CoresPerSocket=24 State=UNKNOWN Feature=skl8168,reconfig
NodeName=els03         Boards=1 SocketsPerBoard=2 CoresPerSocket=24 State=UNKNOWN Feature=skl8160,reconfig
NodeName=els04         Boards=1 SocketsPerBoard=2 CoresPerSocket=20 State=UNKNOWN Feature=skl6148,reconfig
NodeName=els05         Boards=1 SocketsPerBoard=2 CoresPerSocket=28 State=UNKNOWN Feature=skl8180,reconfig
NodeName=enbv01        Boards=1 SocketsPerBoard=2 CoresPerSocket=64 State=UNKNOWN Feature=amd7742a100,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=einsp01       Boards=1 SocketsPerBoard=2 CoresPerSocket=36 State=FUTURE  Feature=icx8360Ya100x8,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=enc101        Boards=1 SocketsPerBoard=2 CoresPerSocket=64 State=FUTURE Feature=amd7763,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=enc[102-109]  Boards=1 SocketsPerBoard=2 CoresPerSocket=64 State=UNKNOWN Feature=amd7763,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=enc110        Boards=1 SocketsPerBoard=2 CoresPerSocket=64 State=FUTURE  Feature=amd7763,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=enc[201-202]  Boards=1 SocketsPerBoard=2 CoresPerSocket=48 State=UNKNOWN Feature=amd7643,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=enc[301-307]  Boards=1 SocketsPerBoard=2 CoresPerSocket=32 State=UNKNOWN Feature=amd7543,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=enc308        Boards=1 SocketsPerBoard=2 CoresPerSocket=32 State=FUTURE  Feature=amd7543,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=enc[309-310]  Boards=1 SocketsPerBoard=2 CoresPerSocket=32 State=UNKNOWN Feature=amd7543,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=enc401        Boards=1 SocketsPerBoard=2 CoresPerSocket=24 State=UNKNOWN Feature=amd74F3,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=enc[501-502]  Boards=1 SocketsPerBoard=2 CoresPerSocket=32 State=UNKNOWN Feature=amd75F3,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
# enc nodes - BIOS settings
NodeName=epb145        Boards=1 SocketsPerBoard=2 CoresPerSocket=24 State=UNKNOWN Feature=clx2s8260L,reconfig,call,oleaf305
NodeName=epb146        Boards=1 SocketsPerBoard=2 CoresPerSocket=24 State=FUTURE Feature=clx2s8260L,reconfig,call,oleaf305
NodeName=epb147        Boards=1 SocketsPerBoard=2 CoresPerSocket=24 State=UNKNOWN Feature=clx2s8260L,reconfig,call,oleaf305
NodeName=epb148        Boards=1 SocketsPerBoard=2 CoresPerSocket=24 State=FUTURE Feature=clx2s8260L,reconfig,call,oleaf305
NodeName=epb[149]  Boards=1 SocketsPerBoard=2 CoresPerSocket=24 State=FUTURE Feature=clx2s8260L,reconfig,call,oleaf305
NodeName=epb[150-152]  Boards=1 SocketsPerBoard=2 CoresPerSocket=24 State=UNKNOWN Feature=clx2s8260L,reconfig,call,oleaf305
NodeName=epb153        Boards=1 SocketsPerBoard=2 CoresPerSocket=24 State=FUTURE  Feature=clx2s8260L,reconfig,call,oleaf305
NodeName=epb[154-156]  Boards=1 SocketsPerBoard=2 CoresPerSocket=24 State=UNKNOWN Feature=clx2s8260L,reconfig,call,oleaf305
NodeName=epb[157-167]  Boards=1 SocketsPerBoard=2 CoresPerSocket=24 State=UNKNOWN Feature=clx2s8260L,reconfig,call,oleaf306
NodeName=epb168  Boards=1 SocketsPerBoard=2 CoresPerSocket=24 State=FUTURE Feature=clx2s8260L,reconfig,call,oleaf306
NodeName=epb[169-177]  Boards=1 SocketsPerBoard=2 CoresPerSocket=24 State=UNKNOWN Feature=clx2s8260L,reconfig,call,oleaf306
NodeName=epb[179-180]  Boards=1 SocketsPerBoard=2 CoresPerSocket=24 State=UNKNOWN Feature=clx2s8260L,reconfig,call,oleaf306
NodeName=epb[181-204]  Boards=1 SocketsPerBoard=2 CoresPerSocket=24 State=UNKNOWN Feature=clx2s8260L,reconfig,call,oleaf307
NodeName=epb[205-228]  Boards=1 SocketsPerBoard=2 CoresPerSocket=24 State=UNKNOWN Feature=clx2s8260L,reconfig,call,oleaf308
NodeName=epb[229-232]  Boards=1 SocketsPerBoard=2 CoresPerSocket=24 State=UNKNOWN Feature=clx2s8260L,reconfig,call,oleaf310
NodeName=epb[235-246]  Boards=1 SocketsPerBoard=2 CoresPerSocket=24 State=UNKNOWN Feature=clx2s8260L,reconfig,call,oleaf310
NodeName=epb[247-252]  Boards=1 SocketsPerBoard=2 CoresPerSocket=24 State=UNKNOWN Feature=clx2s8260L,reconfig,call,oleaf310
NodeName=epb[253-276]  Boards=1 SocketsPerBoard=2 CoresPerSocket=24 State=UNKNOWN Feature=clx2s8260L,reconfig,call,oleaf311
NodeName=epb[277-278]  Boards=1 SocketsPerBoard=2 CoresPerSocket=24 State=UNKNOWN Feature=clx2s8260L,reconfig,call,oleaf312
NodeName=epb279        Boards=1 SocketsPerBoard=2 CoresPerSocket=24 State=FUTURE Feature=clx2s8260L,reconfig,call,oleaf312
NodeName=epb[280-286]  Boards=1 SocketsPerBoard=2 CoresPerSocket=24 State=UNKNOWN Feature=clx2s8260L,reconfig,call,oleaf312
NodeName=epb[288-300]  Boards=1 SocketsPerBoard=2 CoresPerSocket=24 State=UNKNOWN Feature=clx2s8260L,reconfig,call,oleaf312
NodeName=epb[302-308]  Boards=1 SocketsPerBoard=2 CoresPerSocket=24 State=UNKNOWN Feature=clx2s8260L,reconfig,call,oleaf314
NodeName=epb[310-318]  Boards=1 SocketsPerBoard=2 CoresPerSocket=24 State=UNKNOWN Feature=clx2s8260L,reconfig,call,oleaf314
# epb319 removed, need cleanup next PM
NodeName=epb319        Boards=1 SocketsPerBoard=2 CoresPerSocket=24 State=FUTURE  Feature=clx2s8260L,reconfig,call,oleaf314
NodeName=elc[559-560]  Boards=1 SocketsPerBoard=2 CoresPerSocket=16 State=FUTURE  Feature=clx6246R,reconfig
NodeName=elc[601-602]  Boards=1 SocketsPerBoard=2 CoresPerSocket=24 State=FUTURE  Feature=clx8268,reconfig
# HT off in BIOS
#Removed for University Donation NodeName=epb[701-718]    Boards=1 SocketsPerBoard=2 CoresPerSocket=20   State=UNKNOWN Feature=reconfig,clx6230co81

NodeName=elc[801-802]  Boards=1 SocketsPerBoard=2 CoresPerSocket=28 State=FUTURE  Feature=clx8276,reconfig,rocky85
NodeName=elpx01        Boards=1 SocketsPerBoard=4 CoresPerSocket=24 State=UNKNOWN Feature=clx4s8260L
NodeName=elpx02        Boards=1 SocketsPerBoard=4 CoresPerSocket=24 State=FUTURE  Feature=clx4s8260L
NodeName=elpx[03-04]   Boards=1 SocketsPerBoard=4 CoresPerSocket=24 State=UNKNOWN Feature=clx4s8260L

#NodeName=est[01-02]    Boards=1 SocketsPerBoard=2 CoresPerSocket=18 State=UNKNOWN Feature=rebootctrl
#NodeName=est[04,06]    Boards=1 SocketsPerBoard=2 CoresPerSocket=18 State=UNKNOWN Feature=OPA
#NodeName=est[05,07-08] Boards=1 SocketsPerBoard=2 CoresPerSocket=22 State=UNKNOWN Feature=reconfig,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
#NodeName=est[09,10-14] Boards=1 SocketsPerBoard=2 CoresPerSocket=12 State=UNKNOWN Feature=OPA

NodeName=eedc[001-006]  Boards=1 SocketsPerBoard=2 CoresPerSocket=56 State=FUTURE  Feature=dnp56cpq,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eedc[007-012]  Boards=1 SocketsPerBoard=2 CoresPerSocket=56 State=FUTURE  Feature=dnp56cpq,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eedc[013-018]  Boards=1 SocketsPerBoard=2 CoresPerSocket=56 State=FUTURE  Feature=dnp56cpq,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eedc[019-020]  Boards=1 SocketsPerBoard=2 CoresPerSocket=56 State=FUTURE  Feature=dnp56cpq,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8
NodeName=eedc[021-024]  Boards=1 SocketsPerBoard=2 CoresPerSocket=56 State=UNKNOWN Feature=dnp56cpq,corenode,IBSTACK=mlnx-5.5-1.0.3.2_348.12.2.1_2.12.8

# PARTITIONS
PartitionName=adminq    MaxTime=86400 OverSubscribe=YES State=DOWN AllowGroups=crt-dc Nodes=ALL

PartitionName=idealq    MaxTime=86400 OverSubscribe=YES State=DOWN AllowGroups=idealqug    Nodes=eif371,eia[005-013,016,020-042,045,047-053,055-056,060-068,070-071,073,075-078,080-087,089-096,098,100,102-104,106-107,109-114,116,118-123,125-126,145-146,149-154,156-178,180-202,204-211,213,215-216,287-288,503-510,529],eii[017,043-044,046,054,069,097,127-142,144,212,217-235,237,239-277,279-286,308-314,316-321,323-354,379-383]
PartitionName=longq     MaxTime=5760 OverSubscribe=YES State=DOWN AllowGroups=longqug     Nodes=eif371
PartitionName=extq      MaxTime=2880 OverSubscribe=YES State=DOWN AllowGroups=extqug      Nodes=ecx[001,002]
PartitionName=inteldevq MaxTime=5760 OverSubscribe=YES State=DOWN AllowGroups=inteldevqug Nodes=eca[001-028,031-034,038-044,046-056,073-080],ecx[001-004,008],ehx[01-02,04],elb[01-04],elc[02-04],elh[01-04],els[02-05],elc[559-560,601-602,801-802],elpx[01-04],els04,els03,eie[362-369],eid[356-357],eit[289,291,293,295-306],eik[386-389],epb[145-152,154-177,179-232,235-286,288-300,302-308,310-319],eif[372-378,395,396],eil[390-393],eig[358-359],eia[001,003-013,016,020-042,045,047-048,050-053,055-056,060-068,070-071,073,076-087,089-096,098,100-104,106-107,109-114,118-122,124-126,145-146,148-153,155-161,163-175,177,180-186,188-202,204-211,213,501-510,529],eii[002,014-015,017-019,043-044,046,054,057-059,069,072,074,088,097,115,127-144,147,179,212,214,217-225,227-235,237-285,307-314,316-354,379-383],eea[001-040],eim[511-526],eii[324-343]
PartitionName=testq     MaxTime=2880 OverSubscribe=YES State=DOWN AllowGroups=testqug     Nodes=ekln[01-02],eedc[001-024]
PartitionName=egaq      MaxTime=2880 OverSubscribe=YES State=DOWN AllowGroups=egaqug      Nodes=ega[001-016,018-080]
PartitionName=atsq      MaxTime=2880 OverSubscribe=YES State=DOWN AllowGroups=atsqug      Nodes=eii[307-308,311,313-314,316-320,322,324-327,329,331,333-334,336-341,344-345,347-354],eim[511-526],eia529
PartitionName=nextgenq  MaxTime=2880 OverSubscribe=YES State=DOWN AllowGroups=nextgenqug  Nodes=eia[001,003-013,016,020-042,045,047-048,050-053,055-056,060-068,070-071,073,076-087,089-096,098,100-104,106-107,109-114,118-122,124-126,145-146,148-153,155-161,163-175,177,180-186,188-202,204-211,213,501-510,529],eii[002,014-015,017-019,043-044,046,054,057-059,069,072,074,088,097,115,127-144,147,179,212,214,217-225,227-235,237-285,307-314,316-354,379-383],eit[289,291,293,295-306],eea[001-040],eii[324-343]
PartitionName=compq     MaxTime=2880 OverSubscribe=YES State=DOWN AllowGroups=compqug     Nodes=enc[101-110,201,202,301-310,401,501-502]
PartitionName=cnvq      MaxTime=2880 OverSubscribe=YES State=DOWN AllowGroups=cnvqug      Nodes=einv[001-018],einv[527,528]
PartitionName=multigenq MaxTime=2880 OverSubscribe=YES State=DOWN AllowGroups=multigenqug Nodes=ehx[01-02,04],elb[01-04],elc[02-04],elh[01-04],els[02,03-05],elbx[02,03]
PartitionName=workq     MaxTime=2880 OverSubscribe=YES State=DOWN                         Nodes=eca[001-028,031-034,038-044,046-056,073-080],elc[559-560,601-602,801-802],elpx[01-04],ecx[001-004,008],eif[372-378,394-396],eig[358-360],eid[356-357],eie[362-369],eit[289,291,293,295-306],eik[386-389],epb[145-152,154-177,179-232,235-286,288-300,302-308,310-319],eil[390-393],eia[001,003-013,016,020-042,045,047-048,050-053,055-056,060-068,070-071,073,076-087,089-096,098,100-104,106-107,109-114,118-122,124-126,145-146,148-153,155-161,163-175,177,180-186,188-202,204-211,213,501-510,529],eii[002,014-015,017-019,043-044,046,054,057-059,069,072,074,088,097,115,127-144,147,179,212,214,217-225,227-235,237-285,307-314,316-354,379-383]
PartitionName=itarq     MaxTime=2880 OverSubscribe=YES State=DOWN AllowGroups=itarqug     Nodes=ALL
PartitionName=inspurq   MaxTime=2880 OverSubscribe=YES State=DOWN AllowGroups=inspurqug   Nodes=enbv01,einsp01

Comment 3 Scott Hilton 2022-02-10 13:11:45 MST

Can you sent the the output of sinfo?

-Scott

Comment 4 Scott Hilton 2022-02-10 13:43:02 MST

Also do you get the same error when you run the job without the reservation?

Comment 5 Mallick Arigapudi 2022-02-10 13:47:32 MST

(In reply to Scott Hilton from comment #3)
> Can you sent the the output of sinfo?
> 
> -Scott

# sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
adminq       up 60-00:00:0    101   resv eia[012,021,026,028,034,036,041,048-049,060,062,068,075,077-078,080,084,086,089-090,102,110,112,116,123,149,153-154,156-158,160,162,164-165,170,172,176,178,180,182,184,187,196,198-202,206,208-209,211,213,215-216,287-288,503-510,529],eii[017,130,132,134,136-142,217-219,221,223-224,226-227,233-235,246,286,339-343,379-383]
adminq       up 60-00:00:0     47   plnd eca[001-008,010-012,014-019,021-028,031-033,038-043,046-054,056],eia152,eii[230,258]
adminq       up 60-00:00:0      2 drain* eia193,enbv01
adminq       up 60-00:00:0      8  down* eca[009,020],ecx008,eia[076,098,104],eii238,einv013
adminq       up 60-00:00:0    318  alloc eca[013,055],eedc[021-024],eia[005-011,013,016,020,022-025,027,029-033,035,037-040,042,045,047,050-053,055-056,061,063-067,070-071,073,081-083,085,087,091-096,100,103,106-107,109,111,113-114,118-122,125-126,145-146,150-151,155,159,161,163,166-169,171,173-175,177,181,183,185-186,188-192,194-195,197,204-205,207,210],eii[002,014-015,018-019,043-044,046,054,057,059,069,072,074,088,097,115,127-129,131,133,135,144,147,179,212,214,220,222,225,228-229,231-232,237,239-245,247-257,259-277,279-285,308-314,316-321,323-338,344-354],einv[002-003],enc[106-109],epb[145,147,150-152,160-165,167,177,179-217,228,230-232,235-262]
adminq       up 60-00:00:0    271   idle eca[029-030,073-080],ecx[001-004],ega[001-002,004-016,019-080],ehx[01-02,04],eid[356-357],eie[362,367-369],eif[372-375,377-378,394-396],eig358,eik[386-389],eil[390-393],eim[511-512,514-520,524,526],einv[001,005-012,014-017],eit[289-290,292-304,306],elb[01-04],elbx[02-03],elc[02-04],elh[01-04],elpx[01,03-04],els[02-05],enc[102-103,201-202,301-307,309-310,401,501-502],epb[155-159,166,169-176,218-227,229,263-278,280-286,288-300,302-308,310-318]
adminq       up 60-00:00:0      7   down eie[363-366],enc[104-105],epb154
idealq       up 60-00:00:0    101   resv eia[012,021,026,028,034,036,041,048-049,060,062,068,075,077-078,080,084,086,089-090,102,110,112,116,123,149,153-154,156-158,160,162,164-165,170,172,176,178,180,182,184,187,196,198-202,206,208-209,211,213,215-216,287-288,503-510,529],eii[017,130,132,134,136-142,217-219,221,223-224,226-227,233-235,246,286,339-343,379-383]
idealq       up 60-00:00:0      3   plnd eia152,eii[230,258]
idealq       up 60-00:00:0      1 drain* eia193
idealq       up 60-00:00:0      3  down* eia[076,098,104]
idealq       up 60-00:00:0    207  alloc eia[005-011,013,016,020,022-025,027,029-033,035,037-040,042,045,047,050-053,055-056,061,063-067,070-071,073,081-083,085,087,091-096,100,103,106-107,109,111,113-114,118-122,125-126,145-146,150-151,159,161,163,166-169,171,173-175,177,181,183,185-186,188-192,194-195,197,204-205,207,210],eii[043-044,046,054,069,097,127-129,131,133,135,144,212,220,222,225,228-229,231-232,237,239-245,247-257,259-277,279-285,308-314,316-321,323-338,344-354]
longq        up 4-00:00:00      0    n/a
extq         up 2-00:00:00      2   idle ecx[001-002]
inteldevq    up 4-00:00:00     86   resv eia[012,021,026,028,034,036,041,048,060,062,068,077-078,080,084,086,089-090,102,110,112,149,153,156-158,160,164-165,170,172,180,182,184,196,198-202,206,208-209,211,213,503-510,529],eii[017,130,132,134,136-142,217-219,221,223-224,227,233-235,246,339-343,379-383]
inteldevq    up 4-00:00:00     47   plnd eca[001-008,010-012,014-019,021-028,031-033,038-043,046-054,056],eia152,eii[230,258]
inteldevq    up 4-00:00:00      1 drain* eia193
inteldevq    up 4-00:00:00      7  down* eca[009,020],ecx008,eia[076,098,104],eii238
inteldevq    up 4-00:00:00    308  alloc eca[013,055],eia[005-011,013,016,020,022-025,027,029-033,035,037-040,042,045,047,050-053,055-056,061,063-067,070-071,073,081-083,085,087,091-096,100,103,106-107,109,111,113-114,118-122,125-126,145-146,150-151,155,159,161,163,166-169,171,173-175,177,181,183,185-186,188-192,194-195,197,204-205,207,210],eii[002,014-015,018-019,043-044,046,054,057,059,069,072,074,088,097,115,127-129,131,133,135,144,147,179,212,214,220,222,225,228-229,231-232,237,239-245,247-257,259-277,279-285,308-314,316-321,323-338,344-354],epb[145,147,150-152,160-165,167,177,179-217,228,230-232,235-262]
inteldevq    up 4-00:00:00    157   idle eca[073-080],ecx[001-004],ehx[01-02,04],eid[356-357],eie[362,367-369],eif[372-375,377-378,395-396],eig358,eik[386-389],eil[390-393],eim[511-512,514-520,524,526],eit[289,293,295-304,306],elb[01-04],elc[02-04],elh[01-04],elpx[01,03-04],els[02-05],epb[155-159,166,169-176,218-227,229,263-278,280-286,288-300,302-308,310-318]
inteldevq    up 4-00:00:00      5   down eie[363-366],epb154
testq        up 2-00:00:00      4  alloc eedc[021-024]
egaq         up 2-00:00:00     77   idle ega[001-002,004-016,019-080]
atsq         up 2-00:00:00      4   resv eia529,eii[339-341]
atsq         up 2-00:00:00     30  alloc eii[308,311,313-314,316-320,324-327,329,331,333-334,336-338,344-345,347-354]
atsq         up 2-00:00:00     11   idle eim[511-512,514-520,524,526]
nextgenq     up 2-00:00:00     86   resv eia[012,021,026,028,034,036,041,048,060,062,068,077-078,080,084,086,089-090,102,110,112,149,153,156-158,160,164-165,170,172,180,182,184,196,198-202,206,208-209,211,213,503-510,529],eii[017,130,132,134,136-142,217-219,221,223-224,227,233-235,246,339-343,379-383]
nextgenq     up 2-00:00:00      3   plnd eia152,eii[230,258]
nextgenq     up 2-00:00:00      1 drain* eia193
nextgenq     up 2-00:00:00      4  down* eia[076,098,104],eii238
nextgenq     up 2-00:00:00    222  alloc eia[005-011,013,016,020,022-025,027,029-033,035,037-040,042,045,047,050-053,055-056,061,063-067,070-071,073,081-083,085,087,091-096,100,103,106-107,109,111,113-114,118-122,125-126,145-146,150-151,155,159,161,163,166-169,171,173-175,177,181,183,185-186,188-192,194-195,197,204-205,207,210],eii[002,014-015,018-019,043-044,046,054,057,059,069,072,074,088,097,115,127-129,131,133,135,144,147,179,212,214,220,222,225,228-229,231-232,237,239-245,247-257,259-277,279-285,308-314,316-321,323-338,344-354]
nextgenq     up 2-00:00:00     13   idle eit[289,293,295-304,306]
compq        up 2-00:00:00      4  alloc enc[106-109]
compq        up 2-00:00:00     16   idle enc[102-103,201-202,301-307,309-310,401,501-502]
compq        up 2-00:00:00      2   down enc[104-105]
cnvq         up 2-00:00:00      1  down* einv013
cnvq         up 2-00:00:00      2  alloc einv[002-003]
cnvq         up 2-00:00:00     13   idle einv[001,005-012,014-017]
multigenq    up 2-00:00:00     20   idle ehx[01-02,04],elb[01-04],elbx[02-03],elc[02-04],elh[01-04],els[02-05]
workq        up 2-00:00:00     86   resv eia[012,021,026,028,034,036,041,048,060,062,068,077-078,080,084,086,089-090,102,110,112,149,153,156-158,160,164-165,170,172,180,182,184,196,198-202,206,208-209,211,213,503-510,529],eii[017,130,132,134,136-142,217-219,221,223-224,227,233-235,246,339-343,379-383]
workq        up 2-00:00:00     47   plnd eca[001-008,010-012,014-019,021-028,031-033,038-043,046-054,056],eia152,eii[230,258]
workq        up 2-00:00:00      1 drain* eia193
workq        up 2-00:00:00      7  down* eca[009,020],ecx008,eia[076,098,104],eii238
workq        up 2-00:00:00    308  alloc eca[013,055],eia[005-011,013,016,020,022-025,027,029-033,035,037-040,042,045,047,050-053,055-056,061,063-067,070-071,073,081-083,085,087,091-096,100,103,106-107,109,111,113-114,118-122,125-126,145-146,150-151,155,159,161,163,166-169,171,173-175,177,181,183,185-186,188-192,194-195,197,204-205,207,210],eii[002,014-015,018-019,043-044,046,054,057,059,069,072,074,088,097,115,127-129,131,133,135,144,147,179,212,214,220,222,225,228-229,231-232,237,239-245,247-257,259-277,279-285,308-314,316-321,323-338,344-354],epb[145,147,150-152,160-165,167,177,179-217,228,230-232,235-262]
workq        up 2-00:00:00    129   idle eca[073-080],ecx[001-004],eid[356-357],eie[362,367-369],eif[372-375,377-378,394-396],eig358,eik[386-389],eil[390-393],eit[289,293,295-304,306],elpx[01,03-04],epb[155-159,166,169-176,218-227,229,263-278,280-286,288-300,302-308,310-318]
workq        up 2-00:00:00      5   down eie[363-366],epb154
itarq        up 2-00:00:00    101   resv eia[012,021,026,028,034,036,041,048-049,060,062,068,075,077-078,080,084,086,089-090,102,110,112,116,123,149,153-154,156-158,160,162,164-165,170,172,176,178,180,182,184,187,196,198-202,206,208-209,211,213,215-216,287-288,503-510,529],eii[017,130,132,134,136-142,217-219,221,223-224,226-227,233-235,246,286,339-343,379-383]
itarq        up 2-00:00:00     47   plnd eca[001-008,010-012,014-019,021-028,031-033,038-043,046-054,056],eia152,eii[230,258]
itarq        up 2-00:00:00      2 drain* eia193,enbv01
itarq        up 2-00:00:00      8  down* eca[009,020],ecx008,eia[076,098,104],eii238,einv013
itarq        up 2-00:00:00    318  alloc eca[013,055],eedc[021-024],eia[005-011,013,016,020,022-025,027,029-033,035,037-040,042,045,047,050-053,055-056,061,063-067,070-071,073,081-083,085,087,091-096,100,103,106-107,109,111,113-114,118-122,125-126,145-146,150-151,155,159,161,163,166-169,171,173-175,177,181,183,185-186,188-192,194-195,197,204-205,207,210],eii[002,014-015,018-019,043-044,046,054,057,059,069,072,074,088,097,115,127-129,131,133,135,144,147,179,212,214,220,222,225,228-229,231-232,237,239-245,247-257,259-277,279-285,308-314,316-321,323-338,344-354],einv[002-003],enc[106-109],epb[145,147,150-152,160-165,167,177,179-217,228,230-232,235-262]
itarq        up 2-00:00:00    271   idle eca[029-030,073-080],ecx[001-004],ega[001-002,004-016,019-080],ehx[01-02,04],eid[356-357],eie[362,367-369],eif[372-375,377-378,394-396],eig358,eik[386-389],eil[390-393],eim[511-512,514-520,524,526],einv[001,005-012,014-017],eit[289-290,292-304,306],elb[01-04],elbx[02-03],elc[02-04],elh[01-04],elpx[01,03-04],els[02-05],enc[102-103,201-202,301-307,309-310,401,501-502],epb[155-159,166,169-176,218-227,229,263-278,280-286,288-300,302-308,310-318]
itarq        up 2-00:00:00      7   down eie[363-366],enc[104-105],epb154
inspurq      up 2-00:00:00      1 drain* enbv01

Comment 6 Mallick Arigapudi 2022-02-10 13:49:17 MST

(In reply to Scott Hilton from comment #4)
> Also do you get the same error when you run the job without the reservation?

no, it just works fine.

$ bsub -P D:DoD -t 60 -p itarq -C icx8360Y -N 147 -Is sh
interactive
Project Area: D
Project Desc: DoD
srun --pty --exclusive   "-t" "60" "-p" "itarq" "-C" "icx8360Y" "-N" "147" "sh"
srun: environment addon enabled
srun: found Resource_List_CRTDC_PROJECTCODE=D:DoD
srun: job 332041 queued and waiting for resources

Comment 7 Scott Hilton 2022-02-11 10:31:47 MST

Can you send me the slurmctld logs around the time you run the failed command.

Run "scontrol setdebug debug" before you run the command.

Afterwards you can set it back to your slurm.conf setting with "scontrol setdebug info"

-Scott

Comment 8 Scott Hilton 2022-02-16 11:27:36 MST

I have not been able to duplicate your issue. I believe that the debug logs I asked for will help me identify what is going on. 

Also please add these debug flags before reproducing the issue and sending the logs.
scontrol setdebugflags +Reservation
scontrol setdebugflags +SelectType
scontrol setdebugflags +Backfill

-Scott

Comment 9 Jason Booth 2022-02-21 16:12:29 MST

We are lowering the severity pending an update from your site. Please feel free to raise it upon uploading the requested information mentioned previously.

Comment 10 Mallick Arigapudi 2022-03-11 07:20:34 MST

Sorry, lost track of time due to other fires. Will work on this again next week.

Comment 11 Scott Hilton 2022-04-01 14:10:31 MDT

Is this still an issue? Any updates?

-Scott

Comment 12 Scott Hilton 2022-05-16 16:25:19 MDT

Is this still an issue? Any updates?

-Scott

Comment 13 Scott Hilton 2022-06-27 10:54:08 MDT

Closing ticket as timed out. Feel free to reopen the ticket if you have more information on the issue.

-Scott