Created attachment 5189 [details] 0001-mpi-pmix-Implementation-of-support-the-multi-version.patch Hello, This patch introduce PMIx v2 API support - Address the problem with PMIx v2 component architecture (dlopen failure with "undefined symbol"). Detailed problem description: Starting from version 2 PMIx introduces Modular Component Architecture (MCA) borrowed from Open MPI project. MCA assumes that the software consists from the main (base) library and a set of components (separate shared objects) that implement specific features and use common functionality from the main lib. Components are dlopen()’d at the runtime by the main lib based on the component selection logic. Slurm also has its own component infrastructure and (in particular) pmix component is runtime dlopen()ed by the main binary with the RTLD_LOCAL flag. This nested component architectures was creating a conflict situation where all symbols of the PMIx library (“libpmix.so”) loaded along with “mpi/pmix” plugin was visible only in the scope of “mpi_pmix.so” shared object. PMIx components loaded at later stages (thus not as a part of “mpi_pmix.so”) were unable to access the common functions in “libpmix.so”. To avoid Slurm component infrastructure changes but allow explicit control over the dlopen() mode of the “libpmix.so” “mpi/pmix” plugin was restructured to explicitly dlopen() “libpmix.so” with RTLD_GLOBAL flag. “mpi_pmix.so” component is compiled with RPATH set to “libpmix.so” location which allows dlopen() to automatically find it at runtime without full path specification. If “—without-rpath” option is used during Slurm configuration – the path to “libpmix.so” is hardcoded in the plugin. - “mpi/pmix” plugin code was refactored to provide efficient support for multiple PMIx version without code duplications.
*** Ticket 3636 has been marked as a duplicate of this ticket. ***
Boris, after applying this patch nothing works anymore. I have configured Slurm with --with-pmix=/home/da/pmix/1:/home/da/pmix/2 or even with the before working --with-pmix=/home/da/pmix/1 Nothing works. It seems the slurmstepd exits with no sign of failure leaving the srun hung. I don't get a core file and from what I can tell the process exists correctly but doesn't do anything. Running under valgrind doesn't produce anything helpful as well. This is what I get at the end of the slurmd log [2017-10-09T17:58:44.241] [17212.0] debug: mpi type = pmix_v1 [2017-10-09T17:58:44.241] [17212.0] debug: xcgroup_instantiate: cgroup '/sys/fs/cgroup/freezer/slurm_snowflake3' already exists [2017-10-09T17:58:44.241] [17212.0] debug: xcgroup_instantiate: cgroup '/sys/fs/cgroup/freezer/slurm_snowflake3/uid_7558' already exists [2017-10-09T17:58:44.241] [17212.0] debug: spank: opening plugin stack /home/da/slurm/17.11/snowflake/etc/plugstack.conf [2017-10-09T17:58:44.241] [17212.0] debug: mpi type = (null) [2017-10-09T17:58:44.241] [17212.0] debug: (null) [0] /mpi_pmix.c:141 [p_mpi_hook_slurmstepd_prefork] mpi/pmix: start [2017-10-09T17:58:44.241] [17212.0] debug: mpi/pmix: setup sockets Then nothing else. I have tried with pmix_client as well as a simple all2all process with ompi. Both fail. Also just glancing at the code void *libpmix_open(void) { void *lib_plug = NULL; #ifdef PMIXP_V1_LIBPATH char *full_path = NULL; xstrfmtcat(full_path, "%s/libpmix.so", PMIXP_V1_LIBPATH); lib_plug = dlopen("libpmix.so", RTLD_LAZY | RTLD_GLOBAL); xfree(full_path); #elif PMIXP_V2_LIBPATH xstrfmtcat(full_path, "%s/libpmix.so", PMIXP_V2_LIBPATH); lib_plug = dlopen("libpmix.so", RTLD_LAZY | RTLD_GLOBAL); xfree(full_path); #else lib_plug = dlopen("libpmix.so", RTLD_LAZY | RTLD_GLOBAL); #endif if (lib_plug && (HAVE_PMIX_VER != pmixp_lib_get_version())) { PMIXP_ERROR("pmi/pmix: incorrect PMIx library version loaded %d was loaded, required %d version", pmixp_lib_get_version(), (int)HAVE_PMIX_VER); libpmix_close(lib_plug); lib_plug = NULL; } return lib_plug; } Doesn't appear would ever work correctly when PMIXP_V2_LIBPATH is defined, (char *full_path isn't defined, nor is it ever used). I haven't looked much further in the code though. Note I am falling into the 3rd case with since I have rpath installed. It looks like I am loading just fine and get the correct version back from pmixp_lib_get_version(). Any ideas?
Danny, Indeed you right about LIBPATH bug. We need to have full_patch defined correctly. PMIXP_V1_LIBPATH and PMIXP_V2_LIBPATH are used when "--without-rpath" is passed to the Slurm's "./configure". Normally I expect you to go through this codepath: #else lib_plug = dlopen("libpmix.so", RTLD_LAZY | RTLD_GLOBAL); #endif which assumes that rpath wasn't disabled and then during the compilation libpmix should be in the rpath of the pmix plugin binary. In this case according to the dlopen manpage: ... If filename is NULL, then the returned handle is for the main program. If filename contains a slash ("/"), then it is interpreted as a (relative or absolute) pathname. Otherwise, the dynamic linker searches for the object as follows (see ld.so(8) for further details): o (ELF only) If the executable file for the calling program contains a DT_RPATH tag, and does not contain a DT_RUNPATH tag, then the directories listed in the DT_RPATH tag are searched. .... we expect DT_RPATH to be used to search the libpmix.
Boris is going address the bug with LIBPATH you found and update the patch here. HE will also try to reproduce what you are seeing but it seems to me that it is again the same thing: your configuration is somewhat exotic and we haven't hit this problem in our normal setups. From our side I can tell that this patch runs fine in our MTT since we opened this issue (more than a month).
Would you mind attaching your configuration files as well? I can address this simple fix with LIBPATH, no need to resend. If had given me shaky confidence in the rest of the patch though (as this has obviously not been tested). In any case please let me know what you find out. We need this asap now if we want it in 17.11.
Boris will send it to you shortly. We aim to address this today hopefully
Created attachment 5352 [details] slurm.conf
Created attachment 5353 [details] local.conf
Hello, I'm attaching the configuration files that already works in our MTT. Also here are the configs of the builds: Slurm: ./configure --prefix=/sandbox/installs/slurm --with-munge=/usr/ --with-pmix=/sandbox/installs/pmix --with-hwloc=/sandbox/installs/hwloc PMIx: ./configure --prefix=/sandbox/installs/pmix --disable-visibility --enable-debug --with-libevent=/sandbox/installs/libevent --enable-dstore
I don't see much that would matter here outside of the proctrack (we use cgroup), but even then I don't think that would matter either. Boris, will you please attempt to configure Slurm with both pmix_v1 and v2? That would had at least caught the bad code at the beginning. Will you also please try doing this on a multi slurmd system (https://slurm.schedmd.com/faq.html#multi_slurmd)? This is how we do all our testing. This was working before your patch.
I got it to work for v1 now. I will test v2 in a few minutes. It looked like it was all in the loading at the beginning. It claimed it was working, but it wasn't. I have a minor patch fixing that. I'll let you know if I have any more issues going forward.
Created attachment 5354 [details] 0001-mpi-pmix-Implementation-of-support-the-multi-version.patch Updated patch
Danny, I added the updated patch where fixed a macro fork and undefined variable. I checked that occurs a build both versions of PMIx, and both versions are working with "pmix_client --job-fence". Now I'm trying to check as you advised on a multi slurmd system.
Boris, please only send a diff from the first. As I pointed out I am already running with the original. I have already fixed the load at the start. I am now consolidating a host of duplicate code from the 2 versions. Mostly what was done in pmixp_libpmix_job_set, it appears 100% of that code is duplicate.
Created attachment 5358 [details] mpi_pmix.patch Intermediate patch that fixes build issue.
Danny, when testing on a multi slurmd system do you build PMIx with dstore or without?
Created attachment 5359 [details] Fix issue with load/build pmix Boris, this is what I ended up doing to make it work Note: #elif defined PMIXP_V2_LIBPATH was needed. #elif PMIXP_V2_LIBPATH doesn't work.
(In reply to Boris Karasev from comment #16) > Danny, when testing on a multi slurmd system do you build PMIx with dstore > or without? I disable it, but only because Artem told me to. I can try in a bit to see if it works with it. configure --prefix=/home/da/pmix/2 --disable-dstore --disable-visibility
OK, this has been pushed to master along with a couple of clean up commits. c539d34684 5a6aa58f1d 3158332182 d6888cbc85 Please reopen if things are as you would expect.
Boris, I can't seem to get v2.1 to work configure --prefix=/home/da/pmix/2 --disable-visibility and I get this from srun srun --mpi=pmix_v2 -n8 -N1 --exclusive ~/pmix/2/build/test/pmix_client -n 8 --job-fence -c ERROR [pmix_client.c:116:main]: rank 0: PMIx_Get universe size failed: -46 ERROR [pmix_client.c:116:main]: rank 1: PMIx_Get universe size failed: -46 ERROR [pmix_client.c:116:main]: rank 2: PMIx_Get universe size failed: -46 ERROR [pmix_client.c:116:main]: rank 3: PMIx_Get universe size failed: -46 ERROR [pmix_client.c:116:main]: rank 4: PMIx_Get universe size failed: -46 ERROR [pmix_client.c:116:main]: rank 5: PMIx_Get universe size failed: -46 ERROR [pmix_client.c:116:main]: rank 6: PMIx_Get universe size failed: -46 ERROR [pmix_client.c:116:main]: rank 7: PMIx_Get universe size failed: -46 Any ideas?
Created attachment 5669 [details] Fix for the v2.1 problem This patch fixes the observed problem. Boris, please run it through our MTT and report back here. I also fixed possible race condition that was observed on Power platforms by introducing spinlocks around the flag that is used by multiple threads. This was described in https://github.com/pmix/pmix/issues/347.
Also for the reference I fixed the error path logic in PMIx so the error will occur at the place where it is actually happening.
(In reply to Artem Polyakov from comment #24) > Created attachment 5669 [details] > Fix for the v2.1 problem > > This patch fixes the observed problem. > > Boris, please run it through our MTT and report back here. PMIx v2.1.0rc1 was tested with patched Slurm in our MTT. The tests are passed successfully.
Thanks Artem/Boris I have also confirmed this fixes Slurm against v2.1 and continues to work against v2.0. This has been committed (b61fe2ac584c). Thanks!