Ticket 4131

Summary: mpi/pmix: Implementation of support the multi version of PMIx lib
Product: Slurm Reporter: Boris Karasev <karasev.b>
Component: ContributionsAssignee: Danny Auble <dauble>
Status: RESOLVED FIXED QA Contact:
Severity: 4 - Minor Issue    
Priority: --- CC: artpol84, asanchezg, dauble, karasev.b
Version: 17.11.x   
Hardware: Linux   
OS: Linux   
See Also: https://bugs.schedmd.com/show_bug.cgi?id=10291
Site: -Other- Slinky Site: ---
Alineos Sites: --- Atos/Eviden Sites: ---
Confidential Site: --- Coreweave sites: ---
Cray Sites: --- DS9 clusters: ---
Google sites: --- HPCnow Sites: ---
HPE Sites: --- IBM Sites: ---
NOAA SIte: --- NoveTech Sites: ---
Nvidia HWinf-CS Sites: --- OCF Sites: ---
Recursion Pharma Sites: --- SFW Sites: ---
SNIC sites: --- Tzag Elita Sites: ---
Linux Distro: --- Machine Name:
CLE Version: Version Fixed: 17.11.1
Target Release: --- DevPrio: ---
Emory-Cloud Sites: ---
Ticket Depends on:    
Ticket Blocks: 3909    
Attachments: 0001-mpi-pmix-Implementation-of-support-the-multi-version.patch
slurm.conf
local.conf
0001-mpi-pmix-Implementation-of-support-the-multi-version.patch
mpi_pmix.patch
Fix issue with load/build pmix
Fix for the v2.1 problem

Description Boris Karasev 2017-09-04 04:38:44 MDT
Created attachment 5189 [details]
0001-mpi-pmix-Implementation-of-support-the-multi-version.patch

Hello,

This patch introduce PMIx v2 API support

- Address the problem with PMIx v2 component architecture (dlopen failure with "undefined symbol"). 
Detailed problem description: 
Starting from version 2 PMIx introduces Modular Component Architecture (MCA) borrowed from Open MPI project. MCA assumes that the software consists from the main (base) library and a set of components (separate shared objects) that implement specific features and use common functionality from the main lib. Components are dlopen()’d at the runtime by the main lib based on the component selection logic.
Slurm also has its own component infrastructure and (in particular) pmix component is runtime dlopen()ed by the main binary with the RTLD_LOCAL flag.
This nested component architectures was creating a conflict situation where all symbols of the PMIx library (“libpmix.so”) loaded along with “mpi/pmix” plugin was visible only in the scope of “mpi_pmix.so” shared object. PMIx components loaded at later stages (thus not as a part of “mpi_pmix.so”) were unable to access the common functions in “libpmix.so”.
To avoid Slurm component infrastructure changes but allow explicit control over the dlopen() mode of the “libpmix.so” “mpi/pmix” plugin was restructured to explicitly dlopen() “libpmix.so” with RTLD_GLOBAL flag. “mpi_pmix.so” component is compiled with RPATH set to “libpmix.so” location which allows dlopen() to automatically find it at runtime without full path specification. If “—without-rpath” option is used during Slurm configuration – the path to “libpmix.so” is hardcoded in the plugin.

- “mpi/pmix” plugin code was refactored to provide efficient support for multiple PMIx version without code duplications.
Comment 1 Danny Auble 2017-09-25 21:36:31 MDT
*** Ticket 3636 has been marked as a duplicate of this ticket. ***
Comment 2 Danny Auble 2017-10-09 18:22:02 MDT
Boris, after applying this patch nothing works anymore.

I have configured Slurm with --with-pmix=/home/da/pmix/1:/home/da/pmix/2 or even with the before working --with-pmix=/home/da/pmix/1

Nothing works.  It seems the slurmstepd exits with no sign of failure leaving the srun hung.  I don't get a core file and from what I can tell the process exists correctly but doesn't do anything.  Running under valgrind doesn't produce anything helpful as well.

This is what I get at the end of the slurmd log

[2017-10-09T17:58:44.241] [17212.0] debug:  mpi type = pmix_v1
[2017-10-09T17:58:44.241] [17212.0] debug:  xcgroup_instantiate: cgroup '/sys/fs/cgroup/freezer/slurm_snowflake3' already exists
[2017-10-09T17:58:44.241] [17212.0] debug:  xcgroup_instantiate: cgroup '/sys/fs/cgroup/freezer/slurm_snowflake3/uid_7558' already exists
[2017-10-09T17:58:44.241] [17212.0] debug:  spank: opening plugin stack /home/da/slurm/17.11/snowflake/etc/plugstack.conf
[2017-10-09T17:58:44.241] [17212.0] debug:  mpi type = (null)
[2017-10-09T17:58:44.241] [17212.0] debug:  (null) [0] /mpi_pmix.c:141 [p_mpi_hook_slurmstepd_prefork] mpi/pmix: start
[2017-10-09T17:58:44.241] [17212.0] debug:  mpi/pmix: setup sockets

Then nothing else.

I have tried with pmix_client as well as a simple all2all process with ompi.  Both fail.

Also just glancing at the code

void *libpmix_open(void)
{
	void *lib_plug = NULL;
#ifdef PMIXP_V1_LIBPATH
	char *full_path = NULL;
	xstrfmtcat(full_path, "%s/libpmix.so", PMIXP_V1_LIBPATH);
	lib_plug = dlopen("libpmix.so", RTLD_LAZY | RTLD_GLOBAL);
	xfree(full_path);
#elif PMIXP_V2_LIBPATH
	xstrfmtcat(full_path, "%s/libpmix.so", PMIXP_V2_LIBPATH);
	lib_plug = dlopen("libpmix.so", RTLD_LAZY | RTLD_GLOBAL);
	xfree(full_path);
#else
	lib_plug = dlopen("libpmix.so", RTLD_LAZY | RTLD_GLOBAL);
#endif

	if (lib_plug && (HAVE_PMIX_VER != pmixp_lib_get_version())) {
		PMIXP_ERROR("pmi/pmix: incorrect PMIx library version loaded %d was loaded, required %d version",
			    pmixp_lib_get_version(), (int)HAVE_PMIX_VER);
		libpmix_close(lib_plug);
		lib_plug = NULL;
	}

	return lib_plug;
}

Doesn't appear would ever work correctly when PMIXP_V2_LIBPATH is defined, (char *full_path isn't defined, nor is it ever used).  I haven't looked much further in the code though.  Note I am falling into the 3rd case with since I have rpath installed.  It looks like I am loading just fine and get the correct version back from pmixp_lib_get_version().

Any ideas?
Comment 3 Artem Polyakov 2017-10-09 23:31:22 MDT
Danny,

Indeed you right about LIBPATH bug. We need to have full_patch defined correctly.

PMIXP_V1_LIBPATH and PMIXP_V2_LIBPATH are used when "--without-rpath" is passed to the Slurm's "./configure". Normally I expect you to go through this codepath: 
#else
	lib_plug = dlopen("libpmix.so", RTLD_LAZY | RTLD_GLOBAL);
#endif

which assumes that rpath wasn't disabled and then during the compilation libpmix should be in the rpath of the pmix plugin binary. In this case according to the dlopen manpage:

...
If filename is NULL, then the returned handle is for the main
program.  If filename contains a slash ("/"), then it is interpreted
as a (relative or absolute) pathname.  Otherwise, the dynamic linker
searches for the object as follows (see ld.so(8) for further
details):
  o   (ELF only) If the executable file for the calling program
      contains a DT_RPATH tag, and does not contain a DT_RUNPATH tag,
      then the directories listed in the DT_RPATH tag are searched.
....

we expect DT_RPATH to be used to search the libpmix.
Comment 4 Artem Polyakov 2017-10-09 23:47:03 MDT
Boris is going address the bug with LIBPATH you found and update the patch here.

HE will also try to reproduce what you are seeing but it seems to me that it is again the same thing: your configuration is somewhat exotic and we haven't hit this problem in our normal setups. 

From our side I can tell that this patch runs fine in our MTT since we opened this issue (more than a month).
Comment 5 Danny Auble 2017-10-10 06:11:41 MDT
Would you mind attaching your configuration files as well? I can address this simple fix with LIBPATH, no need to resend.  If had given me shaky confidence in the rest of the patch though (as this has obviously not been tested).

In any case please let me know what you find out. We need this asap now if we want it in 17.11.
Comment 6 Artem Polyakov 2017-10-10 06:24:22 MDT
Boris will send it to you shortly.
We aim to address this today hopefully
Comment 7 Boris Karasev 2017-10-10 09:43:35 MDT
Created attachment 5352 [details]
slurm.conf
Comment 8 Boris Karasev 2017-10-10 09:44:02 MDT
Created attachment 5353 [details]
local.conf
Comment 9 Boris Karasev 2017-10-10 09:45:37 MDT
Hello,
I'm attaching the configuration files that already works in our MTT.

Also here are the configs of the builds:
Slurm:
./configure --prefix=/sandbox/installs/slurm --with-munge=/usr/ --with-pmix=/sandbox/installs/pmix --with-hwloc=/sandbox/installs/hwloc

PMIx:
./configure --prefix=/sandbox/installs/pmix --disable-visibility --enable-debug --with-libevent=/sandbox/installs/libevent --enable-dstore
Comment 10 Danny Auble 2017-10-10 10:12:29 MDT
I don't see much that would matter here outside of the proctrack (we use cgroup), but even then I don't think that would matter either.

Boris, will you please attempt to configure Slurm with both pmix_v1 and v2?  That would had at least caught the bad code at the beginning.

Will you also please try doing this on a multi slurmd system (https://slurm.schedmd.com/faq.html#multi_slurmd)?  This is how we do all our testing.  This was working before your patch.
Comment 11 Danny Auble 2017-10-10 11:55:19 MDT
I got it to work for v1 now.  I will test v2 in a few minutes.  It looked like it was all in the loading at the beginning.  It claimed it was working, but it wasn't.  I have a minor patch fixing that.  I'll let you know if I have any more issues going forward.
Comment 12 Boris Karasev 2017-10-10 12:08:45 MDT
Created attachment 5354 [details]
0001-mpi-pmix-Implementation-of-support-the-multi-version.patch

Updated patch
Comment 13 Boris Karasev 2017-10-10 12:17:03 MDT
Danny, I added the updated patch where fixed a macro fork and undefined variable. I checked that occurs a build both versions of PMIx, and both versions are working with "pmix_client --job-fence". Now I'm trying to check as you advised on a multi slurmd system.
Comment 14 Danny Auble 2017-10-10 12:33:49 MDT
Boris, please only send a diff from the first.  As I pointed out I am already running with the original.  I have already fixed the load at the start.  I am now consolidating a host of duplicate code from the 2 versions.  Mostly what was done in pmixp_libpmix_job_set, it appears 100% of that code is duplicate.
Comment 15 Boris Karasev 2017-10-10 12:38:44 MDT
Created attachment 5358 [details]
mpi_pmix.patch

Intermediate patch that fixes build issue.
Comment 16 Boris Karasev 2017-10-10 12:43:37 MDT
Danny, when testing on a multi slurmd system do you build PMIx with dstore or without?
Comment 17 Danny Auble 2017-10-10 12:45:17 MDT
Created attachment 5359 [details]
Fix issue with load/build pmix

Boris, this is what I ended up doing to make it work

Note: #elif defined PMIXP_V2_LIBPATH was needed. #elif PMIXP_V2_LIBPATH doesn't work.
Comment 18 Danny Auble 2017-10-10 12:46:12 MDT
(In reply to Boris Karasev from comment #16)
> Danny, when testing on a multi slurmd system do you build PMIx with dstore
> or without?

I disable it, but only because Artem told me to.  I can try in a bit to see if it works with it.

configure --prefix=/home/da/pmix/2 --disable-dstore --disable-visibility
Comment 19 Danny Auble 2017-10-10 14:32:16 MDT
OK, this has been pushed to master along with a couple of clean up commits.

c539d34684
5a6aa58f1d
3158332182
d6888cbc85

Please reopen if things are as you would expect.
Comment 22 Danny Auble 2017-12-04 11:47:26 MST
Boris, I can't seem to get v2.1 to work

configure --prefix=/home/da/pmix/2 --disable-visibility

and I get this from srun

srun --mpi=pmix_v2 -n8 -N1 --exclusive ~/pmix/2/build/test/pmix_client -n 8 --job-fence -c
ERROR [pmix_client.c:116:main]: rank 0: PMIx_Get universe size failed: -46
ERROR [pmix_client.c:116:main]: rank 1: PMIx_Get universe size failed: -46
ERROR [pmix_client.c:116:main]: rank 2: PMIx_Get universe size failed: -46
ERROR [pmix_client.c:116:main]: rank 3: PMIx_Get universe size failed: -46
ERROR [pmix_client.c:116:main]: rank 4: PMIx_Get universe size failed: -46
ERROR [pmix_client.c:116:main]: rank 5: PMIx_Get universe size failed: -46
ERROR [pmix_client.c:116:main]: rank 6: PMIx_Get universe size failed: -46
ERROR [pmix_client.c:116:main]: rank 7: PMIx_Get universe size failed: -46

Any ideas?
Comment 24 Artem Polyakov 2017-12-04 20:41:15 MST
Created attachment 5669 [details]
Fix for the v2.1 problem

This patch fixes the observed problem.

Boris, please run it through our MTT and report back here.
I also fixed possible race condition that was observed on Power platforms by introducing spinlocks around the flag that is used by multiple threads.

This was described in https://github.com/pmix/pmix/issues/347.
Comment 25 Artem Polyakov 2017-12-04 20:44:06 MST
Also for the reference I fixed the error path logic in PMIx so the error will occur at the place where it is actually happening.
Comment 26 Boris Karasev 2017-12-05 06:22:19 MST
(In reply to Artem Polyakov from comment #24)
> Created attachment 5669 [details]
> Fix for the v2.1 problem
> 
> This patch fixes the observed problem.
> 
> Boris, please run it through our MTT and report back here.

PMIx v2.1.0rc1 was tested with patched Slurm in our MTT. The tests are passed successfully.
Comment 27 Danny Auble 2017-12-05 10:16:49 MST
Thanks Artem/Boris I have also confirmed this fixes Slurm against v2.1 and continues to work against v2.0.

This has been committed (b61fe2ac584c).

Thanks!