Ticket 22289

We are seeing for single node job using HPE slingshot the following error printed in the job output:

srun: error: Unable to create step for job 73324: Error configuring interconnect

It seems this might be related to single_node_vni since the above is outputted when running with:

srun --network=single_node_vni namd3 stmv.namd

If I do not run with --network option I get a traceback from NAMD and particular message:

Reason: OFI::LrtsInit::fi_domain error, for single node use try --network=single_node_vni

Looking at the code I see there is a SwitchParameter with similar name but I may not always want to have it enabled and assume the job setting should be override but the code suggests it checks the config value.  I am not sure whether it is working and suspect it should be reporting an error in the slurmd log but I do not see any errors in the logs which is strange.

So 2 questions:

1. Should single_node_vni be set in SwitchParameters to be able to use it?
2. Should I be seeing errors in the logs?

Our log settings are:

SlurmctldLogFile        = /var/log/slurm/slurmctld.log
SlurmctldSyslogDebug    = (null)
SlurmdLogFile           = (null)
SlurmdSyslogDebug       = (null)
SlurmSchedLogFile       = (null)
SlurmSchedLogLevel      = 0