Ticket 1513 - ordering of hosts in slurm.conf with --enable-multiple-slurmd
Summary: ordering of hosts in slurm.conf with --enable-multiple-slurmd
Status: RESOLVED INFOGIVEN
Alias: None
Product: Slurm
Classification: Unclassified
Component: slurmctld (show other tickets)
Version: 15.08.x
Hardware: Linux Linux
: 4 - Minor Issue
Assignee: David Bigagli
QA Contact:
URL:
Depends on:
Blocks:
 
Reported: 2015-03-09 11:59 MDT by Michael Gutteridge
Modified: 2015-03-10 11:30 MDT (History)
2 users (show)

See Also:
Site: FHCRC - Fred Hutchinson Cancer Research Center
Slinky Site: ---
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
Google sites: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
NoveTech Sites: ---
Nvidia HWinf-CS Sites: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Tzag Elita Sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed:
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---

Attachments
slurm.conf with misordered nodes (50.69 KB, text/plain)
2015-03-10 05:24 MDT, Michael Gutteridge 		Details
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this ticket.
Description Michael Gutteridge 2015-03-09 11:59:45 MDT 
I'm running into an issue in our test environment- to replicate our production cluster, i've set up a cluster using the "multiple slurmd" feature.

It all seems to work well, but it appears that the order in which nodes are declared in slurm.conf matters-  not sure what the design is here, but (for example) if I put a "real" host at the bottom of slurm.conf I get:

slurmctld: error: find_node_record: lookup failure for puck1
slurmctld: error: node_name2bitmap: invalid node specified puck1
slurmctld: fatal: Invalid node names in partition slapshot2

If I move it above all other node definitions, it starts properly.

It's kind of an issue as I'm building these host files with an external tool (puppet templates) and order isn't really a definable thing.

Any hints and/or advice?

Thanks much

Michael
Comment 1 David Bigagli 2015-03-10 05:18:42 MDT 
Hi could you please attach your slurm.conf that shows the problem?

Thanks,
David
Comment 2 Michael Gutteridge 2015-03-10 05:24:30 MDT 
Created attachment 1701 [details]
slurm.conf with misordered nodes
Comment 3 Michael Gutteridge 2015-03-10 05:25:12 MDT 
Ok- attached... when I start slurmctld with this:

mrg@slapshot[~]: sudo /usr/sbin/slurmctld -D
slurmctld: slurmctld version 15.08.0-0pre2 started on cluster slapshot
slurmctld: layouts: no layout to initialize
slurmctld: error: Reconfiguration for node puck1, ignoring!
slurmctld: _parse_part_spec: changing default partition from slapshot to campus
slurmctld: layouts: loading entities/relations information
slurmctld: error: find_node_record: lookup failure for puck1
slurmctld: Recovered state of 406 nodes
slurmctld: Recovered information about 0 jobs
slurmctld: error: find_node_record: lookup failure for puck1
slurmctld: error: node_name2bitmap: invalid node specified puck1
slurmctld: fatal: Invalid node names in partition slapshot


Thanks

M
Comment 4 David Bigagli 2015-03-10 09:55:31 MDT 
Hi, this was fixed in the commit ce32018a28d6b7a. It is available in 15.08.0pre3.
If you update your code to the latest version you will get the fix.

David
Comment 5 David Bigagli 2015-03-10 11:30:46 MDT 
We discovered the issue still exist in the latest code. Sorry for the confusion.
The solution is to use NodeAddr instead of NodeHostName.

Thanks,
David