Ticket 7980

Hi,

We are currently aware of this issue and working out a solution.

Note that this is only happening with libpmi.  libpmi2 does not need to link against libslurmfull.so (or any slurm lib) and doesn't have this problem, but as you may already know Intel MPI 2019 has dropped support for PMI-2.

See where and why we introduced this:
https://bugs.schedmd.com/show_bug.cgi?id=4918#c47

And the talk with open-mpi:
https://github.com/open-mpi/ompi/issues/5145

I guess Intel MPI is having the same issue.

I will inform you when we've committed a definitive solution.

"I guess Intel MPI is having the same issue."

I'm not sure as there is no more "hard" versionning dependency between libpmi
and slurm which weas due in some slurm17 spec files generating /usr/lib64/libslurm.la  and /usr/lib64/libpmi.la for slurm-devel

------------------------------------------------------------------------
  SLURM 17 : no problem with reproducer 

We have also fallen in bug 4918 in slurm17 and suppressing .la files has been enough to suppress  mpi/openmpi dependency versionning because:

The dependency versioning was due to .la files:
grep slurm /usr/lib64/libslurm.la
# libslurm.la - a libtool library file
dlname='libslurm.so.29'
library_names='libslurm.so.29.0.0 libslurm.so.29 libslurm.so' 
old_library='libslurm.a'
# Version information for libslurm.
# grep slurm /usr/lib64/libpmi.la
dependency_libs=' /usr/lib64/libslurm.la -ldl -lpthread'
We had suppressed these .la file.

In slurm 17, libpmi.so is linked with -lslurm, as libslurm.so is the a link with versionned lib the is not direct versionning dependency.
ls -altr  /usr/lib64/libslurm*
1 root root 7383000 Mar 19  2019 /usr/lib64/libslurm.so.32.0.0 1 root root      18 Mar 20  2019 /usr/lib64/libslurm.so.32 -> libslurm.so.32.0.0
1 root root      18 Mar 20  2019 /usr/lib64/libslurm.so -> libslurm.so.32.0.0


IntelMPI + reproducer application does not lead to the segv 

******************************************************************************

Now in slurm 18, there is no depencency versioning of libpmi.so for two reasons no .la files generated with your slurm.spec and you have linked 
libpmi with libslurmfull.so (-lslurmfull). So I don't think linking with -lslurmfull is necessary, the no generation of .la files is enough.

If I rebuild libpmi.so with -lslurm (as in slurm17) there is no dependency versionning between pmi/openmpi as no .la files and the INTELMPI + reproducer does no more SEGV. 

ldd ~gaudin/libpmi.so
    libslurm.so.33 => /lib64/libslurm.so.33 (0x00002ae0e5343000) 

 Sep 30 10:29 /usr/lib64/libslurm.so.33.0.0
1726536 Sep 30 10:29 /usr/lib64/libslurmdb.so.33.0.0
 18 Sep 30 10:33 /usr/lib64/libslurm.so -> libslurm.so.33.0.0


***************************************************************************
            dependency versioning               libpmi.so          reproducer
slurm 17    no (no .la files)                  -lslurm              ok
slurm 18    no (no .la files +? -lslurmfull)   -lslurmfull.so       segv
slurm 18    no (no .la files)                  -lslurm              ok

Thanks

Regine

As the failure seels to be due  to a conjunction of calling debug function of slurmfull.so and debug variable definition in application, I ha

As the failure is a conjunction of calling debug function in slurmfull.so and definition of debug variable in application, I have written another reproducer outside of MPI. I hope it does make sense:

cat hello.c
#include <stdio.h>
#include <slurm.h>
static int my_init_slurm_conf(const char *file_name)
{
        char *name = (char *)file_name;
        int rc =0; 
        debug("Reading slurm.conf file: %s", name);
        return rc;
}

void main(argc,argv)
int argc;
char *argv[];
{
        if (my_init_slurm_conf("/etc/slurm/slurm.conf") == 0) 
            printf("read succeed \n");
        else
            printf("read failed \n");
}

icc -g -o hello hello.c  -I/usr/include/slurm -L/usr/lib64/slurm -lslurmfull

ldd hello
	linux-vdso.so.1 =>  (0x00007ffdcd968000)
	libslurmfull.so => /usr/lib64/slurm/libslurmfull.so (0x00007f162a9d9000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f162a6c9000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f162a4b3000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f162a0f2000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f1629eed000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f1629cd1000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f162ada5000)

./hello
read succeed


Now if I link with just a c file defining debug variable it segvs in debug call

cat just_define_debug.c
int debug = 0;

icc -fPIC -c -g just_define_debug.c
icc -shared just_define_debug.o -o ./libsup7intel.so

 icc -g -o hello hello.c just_define_debug.c -I/usr/include/slurm -L/usr/lib64/slurm -lslurmfull -L$PWD -lsup7intel

 ldd hello
	linux-vdso.so.1 =>  (0x00007ffd027e2000)
	libslurmfull.so => /usr/lib64/slurm/libslurmfull.so (0x00007fdb4ec2b000)
	libsup7intel.so => /ccc/home/cont001/ocre/gaudinr/debug_so/libsup7intel.so (0x00007fdb4ea28000)

./hello
Segmentation fault

(gdb) where
#0  0x0000000000601030 in debug ()
#1  0x0000000000400674 in my_init_slurm_conf (file_name=0x40078c "/etc/slurm/slurm.conf") at hello.c:7
#2  0x0000000000400698 in main (argc=1, argv=0x7fffffffe1a8) at hello.c:15

If I rename debug variable in debug2 in works fine

cat just_define_debug.c
int debug2 = 0;

 icc -fPIC -c -g just_define_debug.c
(=
 $ icc -shared just_define_debug.o -o ./libsup7intel.so

$  icc -g -o hello hello.c just_define_debug.c -I/usr/include/slurm -L/usr/lib64/slurm -lslurmfull -L$PWD -lsup7intel

 ./hello
read succeed

I have to study more your comments, but the fact we are linking with the .la files is because OpenMPI (i.e. libtool) uses rpath/runpath linking. If the .la is removed from the system one will need to set LD_LIBRARY_PATH manually because rpath won't be set in the elf file. This situation is undesirable since most people is not installing the softwares in standard paths, like you seem to be doing (/usr/lib64/).

You can read through https://github.com/open-mpi/ompi/issues/5145

See my comment here suggesting to remove .la files and further responses:
https://github.com/open-mpi/ompi/issues/5145#issuecomment-387350602

The other solution we are working on is to create a new API for pmi and make others link to this unversioned one, but we're still working on it.

I will work on this issue more tomorrow.

Thanks

Hi,

That's just a note to confirm that we're working on this specific issue in the other internal bug 7980.

I will let you know when it is fixed, or if you prefer I can close this issue and you will be automatically notified when bug 7980 is resolved.

Hi
Keep this one opened and notify when internal 7980 will be fixed.
Thanks
Regine

Just a note that we (Compute Canada, where the Cedar cluster recently upgraded to Slurm 19) have also been bitten by this bug but using Open MPI instead. If Open MPI is compiled with --disable-dlopen so a compiled application directly links to libpmi.so it's even worse and can be reproduced using the simple:

#include <mpi.h>
int debug;
int main(int argc, char **argv) {
        MPI_Init(&argc, &argv);
        MPI_Finalize();
        return 0;
}

We initially found it to crash with Gromacs 5.1.4, which also defines a "debug" variable.

Of course, as mentioned, with OpenMPI, using PMIx or libpmi2 avoids the issue, but still the underlying issue is that libslurmfull.so makes all its symbols global, unlike libslurm.so which only exports symbols following src/api/version.map:
{ global:
   islurm_*;
   slurm_*;
   slurmdb_*;
   plugin_context_*;
   working_cluster_rec;
  local: *;
};

so it seems to me the cure for fixing the .la versioning annoyance is worse than the disease...

Increase priority as another application is failing in this bug.

War replacing debug by slurm_debug in _init_slurm_conf function is in a process
of validation

(In reply to Regine Gaudin from comment #9)
> Increase priority as another application is failing in this bug.
> 
> War replacing debug by slurm_debug in _init_slurm_conf function is in a
> process
> of validation

Hi, I am trying to get the patch which fixes this issue commited in upstream ASAP.

Will let you know, and will see if I can provide it to you as a pre-patch.

(In reply to Felip Moll from comment #10)
> (In reply to Regine Gaudin from comment #9)
> > Increase priority as another application is failing in this bug.
> > 
> > War replacing debug by slurm_debug in _init_slurm_conf function is in a
> > process
> > of validation
> 
> Hi, I am trying to get the patch which fixes this issue commited in upstream
> ASAP.
> 
> Will let you know, and will see if I can provide it to you as a pre-patch.

Hi Regine,

I am trying to push for commiting the patch which will fix the issue in bug 7980.
I just only wanted to let you know I haven't forgot about it.

It's quite a big modification so I prefer first it to be reviewed before giving it to you directly.

Sorry for the inconvenience and long delay.

Note that in my all previous comments I always meant 'the private bug 7448'.

Bug 7980 is this one.

Hi,

I just want to inform you again that bug 7448 is in our internal review process and when the proposed patch is accepted by the QA team, it will fix your issues.

I'll inform when it is done.

Thanks for your patience and sorry for being suffering this delay.

Hi,

(In reply to Felip Moll from comment #13)
> I just want to inform you again that bug 7448 is in our internal review
> process and when the proposed patch is accepted by the QA team, it will fix
> your issues.
> 
> I'll inform when it is done.

Any news about this?

We are facing the same bug at IDRIS (supercomputing center in France).

Best regards,
Rémi Lacroix

(In reply to Rémi Lacroix from comment #14)
> Hi,
> 
> (In reply to Felip Moll from comment #13)
> > I just want to inform you again that bug 7448 is in our internal review
> > process and when the proposed patch is accepted by the QA team, it will fix
> > your issues.
> > 
> > I'll inform when it is done.
> 
> Any news about this?
> 
> We are facing the same bug at IDRIS (supercomputing center in France).
> 
> Best regards,
> Rémi Lacroix

Hi Rémi,

I don't have any news. We're in the middle of QA for 20.02 release and bug 7448 hasn't been considered yet for review. I'll inform when it is.

I don't have you in our contact list, are you working with Philippe Collinet and are authorized to open/interact with IDRIS bugs?

Thanks

(In reply to Felip Moll from comment #15)
> I don't have any news. We're in the middle of QA for 20.02 release and bug
> 7448 hasn't been considered yet for review. I'll inform when it is.

Ok, any chance for it be backported to the 19.X branch? I am not sure we are planning to upgrade to 20.X in the near future.

> I don't have you in our contact list, are you working with Philippe Collinet
> and are authorized to open/interact with IDRIS bugs?

I am working with Philippe but I'm not a sysadmin at IDRIS, I'm part of the user support team (so working on MPI, HPC codes, that sort of things). Philippe usually open the bugs but I interact directly with them afterwards (or when they exist already like this one).

Rémi

> Ok, any chance for it be backported to the 19.X branch? I am not sure we are
> planning to upgrade to 20.X in the near future.

Not officially, but you'll probably be able to backport it.

Hi, finally the issue has been fixed and you shouldn't experience this anymore in versions starting at 20.11 (current master).


commit d3585a55d5820ee7ae0108d0b730ce9e9be661f8
Author:     Broderick Gardner <broderick@schedmd.com>
AuthorDate: Thu Sep 19 11:14:04 2019 -0600

    Add libslurm_pmi.so as unversioned lib.
    
    Allows libpmi.so to link to an unversioned slurm lib, which will
    help avoid issues when statically compiling OpenMPI.
    
    Bug 7448.