Ticket 5323

Summary:	OpenMPI internal PMIx with Slurm
Product:	Slurm	Reporter:	Jed storey <jstorey2009>
Component:	Build System and Packaging	Assignee:	Jacob Jenson <jacob>
Status:	RESOLVED INVALID	QA Contact:
Severity:	6 - No support contract
Priority:	---	CC:	joshua.schwartz, sts
Version:	17.11.7
Hardware:	Linux
OS:	Linux
See Also:	https://bugs.schedmd.com/show_bug.cgi?id=8625
Site:	-Other-	Alineos Sites:	---
Atos/Eviden Sites:	---	Confidential Site:	---
Coreweave sites:	---	Cray Sites:	---
DS9 clusters:	---	HPCnow Sites:	---
HPE Sites:	---	IBM Sites:	---
NOAA SIte:	---	NoveTech Sites:	---
Nvidia HWinf-CS Sites:	---	OCF Sites:	---
Recursion Pharma Sites:	---	SFW Sites:	---
SNIC sites:	---	Tzag Elita Sites:	---
Linux Distro:	---	Machine Name:
CLE Version:		Version Fixed:
Target Release:	---	DevPrio:	---
Emory-Cloud Sites:	---

Description Jed storey 2018-06-16 06:26:44 MDT

Hi,

OS: CentOS 7.5
MPI: OpenMPI v3.1.0 configured --with-slurm
Slurm: 17.11.7-1

Does Slurm support the use of OpenMPI's internal PMIx v2? The slurm mpi guide seems to suggest so, "Starting with Open MPI version 3.1, PMIx version 2 is natively supported. To launch Open MPI application using PMIx version 2 the '--mpi=pmix_v2' option must be specified on the srun command line". However, I cannot get slurm to build with the openmpi internal pmix installation. I used the "--define '_with_pmix --with-pmix=/opt/openmpi-3.1.0'" rpmbuild option to try to point slurm at the openmpi installation folder, the opal subfolder, and the pmix subfolder, but the slurm rpmbuild always returns with "WARNING: unable to locate pmix installation". The PMIx instructions on the slurm mpi guide page are only for building an external PMIx and then building slurm pointing at that external pmix installation. I suppose I could build both OpenMPI and Slurm to point at the external pmix installation, but I'd rather use the internal openmpi pmix2 if possible.

Currently, the srun --mpi=list command just comes back with pmi2, none, and openmpi. Using srun with either pmi2 or openmpi results in the error stating that either slurm or openmpi were not configured with pmi correctly. Directly launching mpirun from within an sbatch script works fine, though. I think if I configure openmpi --with-pmi2 to point at the slurm pmi2 installation, then the pmi2 option will probably work, so that's my backup plan. Another question: What is the srun --mpi=openmpi option and what is it supposed to do? I can't find documentation for that.

Also, clearly the openmpi configure --with-slurm option doesn't seem to make openmpi inherently work with slurm. That option is not well documented either. I know it's probably a question for openmpi people, but what does the openmpi --with-slurm configure option do? The FAQ page on it is short and unhelpful, and says you have to use the --with-pmi configure option pointing at the slurm pmi directory, so what's the point of the --with-slurm option?

So in summary, I have three questions/issues/maybe-bugs-but-probably-me-doing-something-wrong:
1. Does Slurm support the use of OpenMPI's internal PMIx v2? If yes, how configure slurm build? If no, will it in future versions?
2. What is the srun --mpi=openmpi option and what is it supposed to do?
3. What does the openmpi --with-slurm configure option do? (possibly not right place to ask this)

Thanks

Comment 1 Jed storey 2018-06-16 14:44:58 MDT

I can confirm slurm 17.11.7 and openmpi 3.1.0 work with the srun --mpi=pmi2 option. To get this to work, I installed all slurm rpms on my headnode and all but the db and ctl rpms on a compute node. Then I reconfigured and reinstalled OpenMPI with the following options: "--prefix=/opt/openmpi-3.1.0 --with-verbs --with-slurm --with-pmi=/usr". Oddly, using the "-with-slurm --with-pmi=/usr/include/slurm --with-pmi-libdir=/usr/" didn't work because it couldn't find the libpmi files. There is current an openmpi bug thread about this, and it's definitely an openmpi problem, not a slurm problem. Anyways, srun --mpi=pmi2 now works, which is good. 

I still can't find answers to my previous questions, though.

Comment 2 Josh Schwartz 2019-08-22 11:51:19 MDT

I'm having many of the exact same issues with this.

I can't get OpenMPI to build properly against an external PMIx because it decides that its internal version is newer.  But then I can't get SLURM to build against that same PMIx version because the OpenMPI RPM install doesn't include the pmix_common.h for its internal PMIx.

This is a real mess and it shouldn't be so difficult to get SLURM, OpenMPI, and PMIx all installed with compatible versions.

The closest I got was with using the --with-pmix options on both SLURM and OpenMPI, but then I get segfaults while running which google-fu seems to indicate are from mismatched versions (which isn't a surprise since OpenMPI discards my --with-pmix option in favor of its internal version):

[hostname:177125] *** Process received signal ***
[hostname:177125] Signal: Segmentation fault (11)
[hostname:177125] Signal code: Invalid permissions (2)
[hostname:177125] Failing at address: 0xa3fc28
[hostname:177125] [ 0] /usr/lib64/libpthread.so.0(+0xf6d0)[0x7fd7da0926d0]
[hostname:177125] [ 1] [0xa3fc28]
[hostname:177125] *** End of error message ***

Comment 3 Josh Schwartz 2019-08-22 16:00:33 MDT

The magic for me to finally get this working:

the following options to the OpenMPI rpmbuild:

rpmbuild \
  --define "configure_options --with-pmix=${PMIX} --with-libevent=/usr --with-hwloc=/usr --with-ompi-pmix-rte --with-slurm"
  -ba openmpi.spec

then the following options to the SLURM rpmbuild:

rpmbuild \
  --define "_with_pmix --with-pmix=${PMIX}" \
  -ba slurm.spec

This document was the most helpful:
https://pmix.org/code/building-the-pmix-reference-server/

Make sure your hwloc-devel and libevent-devel package versions are acceptable based on: https://pmix.org/code/getting-the-pmix-reference-server/

Comment 4 Josh Schwartz 2019-08-22 16:03:28 MDT

Also might be worth looking at a couple other issues I had to work around which may affect you depending on what you're doing:

OpenMPI:
https://github.com/open-mpi/ompi/issues/6900
https://github.com/open-mpi/ompi/issues/6914

SLURM:
https://bugs.schedmd.com/show_bug.cgi?id=7584