10193 – scontrol: error: plugin_load_from_file: dlopen(/usr/local/lib/slurm/auth_munge.so): /usr/local/lib/slurm/auth_munge.so: undefined symbol: slurm_conf

Ticket 10193 - scontrol: error: plugin_load_from_file: dlopen(/usr/local/lib/slurm/auth_munge.so): /usr/local/lib/slurm/auth_munge.so: undefined symbol: slurm_conf

Summary: scontrol: error: plugin_load_from_file: dlopen(/usr/local/lib/slurm/auth_mung...

Status:	RESOLVED FIXED

Alias:	None

Product:	Slurm
Classification:	Unclassified
Component:	Build System and Packaging (show other tickets)
Version:	20.11.x
Hardware:	Linux Linux

Severity:	4 - Minor Issue
Assignee:	Nate Rini
QA Contact:

URL:

Depends on:
Blocks:

Reported:	2020-11-11 00:48 MST by Torkil Svensgaard
Modified:	2020-11-30 13:07 MST (History)
CC List:	2 users (show)

See Also:
Site:	DRCMR
Slinky Site:	---
Alineos Sites:	---
Atos/Eviden Sites:	---
Confidential Site:	---
Coreweave sites:	---
Cray Sites:	---
DS9 clusters:	---
Google sites:	---
HPCnow Sites:	---
HPE Sites:	---
IBM Sites:	---
NOAA SIte:	---
NoveTech Sites:	---
Nvidia HWinf-CS Sites:	---
OCF Sites:	---
Recursion Pharma Sites:	---
SFW Sites:	---
SNIC sites:	---
Tzag Elita Sites:	---
Linux Distro:	CentOS
Machine Name:	slurm
CLE Version:
Version Fixed:	20.11
Target Release:	---
DevPrio:	---
Emory-Cloud Sites:	---

Attachments
Debug (669.18 KB, application/x-sharedlib) 2020-11-13 00:09 MST, Torkil Svensgaard	Details
objdump -T (117.85 KB, text/plain) 2020-11-13 00:09 MST, Torkil Svensgaard	Details
objdump -t (1.31 MB, text/plain) 2020-11-13 00:10 MST, Torkil Svensgaard	Details
nm libslurm (669.18 KB, application/x-sharedlib) 2020-11-13 00:10 MST, Torkil Svensgaard	Details
nm slurmfull.so (257.67 KB, text/plain) 2020-11-13 10:22 MST, Torkil Svensgaard	Details
config.log (23.83 KB, application/zip) 2020-11-14 00:56 MST, Torkil Svensgaard	Details
nm libslurmfull.so (669.18 KB, text/plain) 2020-11-17 00:32 MST, Torkil Svensgaard	Details
sym.log (351.11 KB, text/plain) 2020-11-17 14:55 MST, Torkil Svensgaard	Details
Sym2.log (397.41 KB, text/plain) 2020-11-19 00:07 MST, Torkil Svensgaard	Details
slurmctld.log (448.47 KB, text/x-log) 2020-11-19 03:37 MST, Torkil Svensgaard	Details
sym3_with_f (250.07 KB, text/plain) 2020-11-19 23:34 MST, Torkil Svensgaard	Details
sym3 (250.07 KB, text/plain) 2020-11-19 23:34 MST, Torkil Svensgaard	Details
sym4.log - which slurmctld (259.49 KB, text/plain) 2020-11-20 13:07 MST, Torkil Svensgaard	Details
sym3v2 (2.30 KB, text/plain) 2020-11-30 12:09 MST, Torkil Svensgaard	Details
slurmctld.log (354.30 KB, application/gzip) 2020-11-30 12:11 MST, Torkil Svensgaard	Details
Show Obsolete (1) Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this ticket.

Description Torkil Svensgaard 2020-11-11 00:48:02 MST

Built 20.11 with rpmbuild and upgraded from 20.02.5. Works on submit and compute nodes but on the master I get the following error with slurm commands:

"
# scontrol reconfigure
scontrol: error: plugin_load_from_file: dlopen(/usr/local/lib/slurm/auth_munge.so): /usr/local/lib/slurm/auth_munge.so: undefined symbol: slurm_conf
scontrol: error: Couldn't load specified plugin name for auth/munge: Dlopen of plugin file failed
scontrol: error: cannot create auth context for auth/munge
scontrol: error: slurm_send_node_msg: authentication: No such process
scontrol: error: _fetch_child: failed to fetch remote configs
"

Mvh.

Torkil

Comment 1 Nate Rini 2020-11-11 09:25:52 MST

(In reply to Torkil Svensgaard from comment #0)
> Built 20.11 with rpmbuild and upgraded from 20.02.5. Works on submit and
> compute nodes but on the master I get the following error with slurm
> commands:

Make sure to do a clean install. Looks like the auth_munge.so getting linked is the library built for 20.02 (or earlier).

Comment 2 Torkil Svensgaard 2020-11-12 00:22:04 MST

(In reply to Nate Rini from comment #1)
> (In reply to Torkil Svensgaard from comment #0)
> > Built 20.11 with rpmbuild and upgraded from 20.02.5. Works on submit and
> > compute nodes but on the master I get the following error with slurm
> > commands:
> 
> Make sure to do a clean install. Looks like the auth_munge.so getting linked
> is the library built for 20.02 (or earlier).

So dnf upgrade is not a valid way to upgrade slurm?

Mvh.

Torkil

Comment 3 Torkil Svensgaard 2020-11-12 00:33:38 MST

Tried removing and reinstalling slurm packages, same thing.

"
sinfo: error: plugin_load_from_file: dlopen(/usr/local/lib/slurm/auth_munge.so): /usr/local/lib/slurm/auth_munge.so: undefined symbol: slurm_conf
sinfo: error: Couldn't load specified plugin name for auth/munge: Dlopen of plugin file failed
sinfo: error: cannot create auth context for auth/munge
sinfo: error: slurm_send_node_msg: authentication: No such process
sinfo: error: _fetch_child: failed to fetch remote configs
"

"
# ls -lah /usr/local/lib/slurm/auth_munge.so
lrwxrwxrwx. 1 root root 30 Oct 27 13:27 /usr/local/lib/slurm/auth_munge.so -> /usr/lib64/slurm/auth_munge.so

# ls -lah /usr/lib64/slurm/auth_munge.so 
-rwxr-xr-x. 1 root root 239K Nov 11 08:27 /usr/lib64/slurm/auth_munge.so
"

Mvh.

Torkil

Comment 4 Nate Rini 2020-11-12 10:13:48 MST

(In reply to Torkil Svensgaard from comment #2)
> So dnf upgrade is not a valid way to upgrade slurm?
dnf install of rpms should work as long as you're replacing all of the previous rpms (and not forcing an install).

Please note that upgrades using dnf/rpm still must follow the normal upgrade order as specified in Upgrades under:
> https://slurm.schedmd.com/quickstart_admin.html

Also, slurm-20.11 is a release candidate and should not be used in a production environment. Your testing of release candidates helps us catch issues that may have been missed by our testing.

(In reply to Torkil Svensgaard from comment #3)
> Tried removing and reinstalling slurm packages, same thing.
>
> # ls -lah /usr/lib64/slurm/auth_munge.so 
> -rwxr-xr-x. 1 root root 239K Nov 11 08:27 /usr/lib64/slurm/auth_munge.so

It is possible there is some issue with the rpm specification. Can you please call the following to allow us to verify the installed library:
> nm /usr/lib64/slurm/auth_munge.so
> ldd /usr/lib64/slurm/auth_munge.so
> objdump -T /usr/lib64/slurm/auth_munge.so
> objdump -t /usr/lib64/slurm/auth_munge.so

> nm /usr/lib64/libslurm.so #path to libslurm.so may need to be corrected
> ldd /usr/lib64/libslurm.so
> objdump -T /usr/lib64/libslurm.so
> objdump -t /usr/lib64/libslurm.so

Comment 5 Torkil Svensgaard 2020-11-13 00:09:06 MST

Created attachment 16619 [details]
Debug

Comment 6 Torkil Svensgaard 2020-11-13 00:09:30 MST

(In reply to Nate Rini from comment #4)
> (In reply to Torkil Svensgaard from comment #2)
> > So dnf upgrade is not a valid way to upgrade slurm?
> dnf install of rpms should work as long as you're replacing all of the
> previous rpms (and not forcing an install).

I believe that is what I am doing.
 
> (In reply to Torkil Svensgaard from comment #3)
> > Tried removing and reinstalling slurm packages, same thing.
> >
> > # ls -lah /usr/lib64/slurm/auth_munge.so 
> > -rwxr-xr-x. 1 root root 239K Nov 11 08:27 /usr/lib64/slurm/auth_munge.so
> 
> It is possible there is some issue with the rpm specification. Can you
> please call the following to allow us to verify the installed library:

Sure

> > nm /usr/lib64/slurm/auth_munge.so

# nm /usr/lib64/slurm/auth_munge.so
00000000000013f9 t .annobin_auth_munge.c
0000000000001e3a t .annobin_auth_munge.c_end
0000000000001340 t .annobin_auth_munge.c_end.exit
0000000000001340 t .annobin_auth_munge.c_end.hot
0000000000001340 t .annobin_auth_munge.c_end.startup
0000000000001340 t .annobin_auth_munge.c_end.unlikely
0000000000001340 t .annobin_auth_munge.c.exit
0000000000001340 t .annobin_auth_munge.c.hot
0000000000001340 t .annobin_auth_munge.c.startup
0000000000001340 t .annobin_auth_munge.c.unlikely
000000000000160b t .annobin_init.end
0000000000001590 t .annobin_init.start
0000000000001590 t .annobin__print_cred.end
00000000000013f9 t .annobin__print_cred.start
00000000000018a6 t .annobin_slurm_auth_create.end
000000000000160b t .annobin_slurm_auth_create.start
00000000000018ea t .annobin_slurm_auth_destroy.end
00000000000018a6 t .annobin_slurm_auth_destroy.start
0000000000001b92 t .annobin_slurm_auth_get_gid.end
0000000000001b6a t .annobin_slurm_auth_get_gid.start
0000000000001ca0 t .annobin_slurm_auth_get_host.end
0000000000001b92 t .annobin_slurm_auth_get_host.start
0000000000001b6a t .annobin_slurm_auth_get_uid.end
0000000000001b42 t .annobin_slurm_auth_get_uid.start
0000000000001d24 t .annobin_slurm_auth_pack.end
0000000000001ca0 t .annobin_slurm_auth_pack.start
0000000000001e30 t .annobin_slurm_auth_thread_clear.end
0000000000001e2b t .annobin_slurm_auth_thread_clear.start
0000000000001e2b t .annobin_slurm_auth_thread_config.end
0000000000001e21 t .annobin_slurm_auth_thread_config.start
0000000000001e3a t .annobin_slurm_auth_token_generate.end
0000000000001e30 t .annobin_slurm_auth_token_generate.start
0000000000001e21 t .annobin_slurm_auth_unpack.end
0000000000001d24 t .annobin_slurm_auth_unpack.start
0000000000001b42 t .annobin_slurm_auth_verify.end
00000000000018ea t .annobin_slurm_auth_verify.start
0000000000203100 d bad_cred_test
0000000000203104 B __bss_start
0000000000203104 b completed.7294
                 w __cxa_finalize@@GLIBC_2.2.5
0000000000001340 t deregister_tm_clones
00000000000013b0 t __do_global_dtors_aux
0000000000202d98 t __do_global_dtors_aux_fini_array_entry
0000000000202da0 d __dso_handle
0000000000202da8 d _DYNAMIC
0000000000203104 D _edata
0000000000203108 B _end
0000000000001e3c t _fini
00000000000013f0 t frame_dummy
0000000000202d90 t __frame_dummy_init_array_entry
00000000000023e0 r __FRAME_END__
                 U free@@GLIBC_2.2.5
0000000000002122 r __func__.17415
0000000000002110 r __func__.17427
00000000000020d0 r __func__.17451
00000000000020c0 r __func__.17458
00000000000020a0 r __func__.17465
00000000000020f8 r __func__.17475
00000000000020e8 r __func__.17484
                 U getenv@@GLIBC_2.2.5
                 U get_name_info
0000000000203000 d _GLOBAL_OFFSET_TABLE_
                 w __gmon_start__
000000000000215c r __GNU_EH_FRAME_HDR
0000000000000f70 t _init
0000000000001590 T init
                 w _ITM_deregisterTMCloneTable
                 w _ITM_registerTMCloneTable
                 U munge_ctx_create
                 U munge_ctx_destroy
                 U munge_ctx_get
                 U munge_ctx_set
                 U munge_ctx_strerror
                 U munge_decode
                 U munge_encode
000000000000212c R plugin_id
0000000000002140 R plugin_name
0000000000002130 R plugin_type
0000000000002128 R plugin_version
00000000000013f9 t _print_cred
0000000000001370 t register_tm_clones
000000000000160b T slurm_auth_create
00000000000018a6 T slurm_auth_destroy
0000000000001b6a T slurm_auth_get_gid
0000000000001b92 T slurm_auth_get_host
0000000000001b42 T slurm_auth_get_uid
                 U slurm_auth_opts_to_socket
0000000000001ca0 T slurm_auth_pack
0000000000001e2b T slurm_auth_thread_clear
0000000000001e21 T slurm_auth_thread_config
0000000000001e30 T slurm_auth_token_generate
0000000000001d24 T slurm_auth_unpack
00000000000018ea T slurm_auth_verify
                 U slurm_conf
                 U slurm_ctime2_r
                 U slurm_error
                 U slurm_get_auth_ttl
                 U slurm_get_ip_str
                 U slurm_get_log_level
                 U slurm_log_var
                 U slurm_packmem
                 U slurm_seterrno
                 U slurm_unpackmem_malloc
                 U slurm_xcalloc
                 U slurm_xfree
                 U slurm_xsignal
                 U __stack_chk_fail@@GLIBC_2.4
                 U strtol@@GLIBC_2.2.5
                 U time@@GLIBC_2.2.5
0000000000203108 d __TMC_END__
                 U usleep@@GLIBC_2.2.5

> > ldd /usr/lib64/slurm/auth_munge.so

# ldd /usr/lib64/slurm/auth_munge.so
	linux-vdso.so.1 (0x00007ffe14ea3000)
	libmunge.so.2 => /usr/lib64/libmunge.so.2 (0x00007f56385a3000)
	libm.so.6 => /usr/lib64/libm.so.6 (0x00007f5638221000)
	libresolv.so.2 => /usr/lib64/libresolv.so.2 (0x00007f563800a000)
	libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x00007f5637dea000)
	libc.so.6 => /usr/lib64/libc.so.6 (0x00007f5637a28000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f56389b1000)

> > objdump -T /usr/lib64/slurm/auth_munge.so

# objdump -T /usr/lib64/slurm/auth_munge.so

/usr/lib64/slurm/auth_munge.so:     file format elf64-x86-64

DYNAMIC SYMBOL TABLE:
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.2.5 getenv
0000000000000000      DF *UND*	0000000000000000              munge_decode
0000000000000000      D  *UND*	0000000000000000              slurm_get_log_level
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.2.5 free
0000000000000000      DF *UND*	0000000000000000              munge_ctx_destroy
0000000000000000      D  *UND*	0000000000000000              slurm_get_auth_ttl
0000000000000000  w   D  *UND*	0000000000000000              _ITM_deregisterTMCloneTable
0000000000000000      D  *UND*	0000000000000000              slurm_unpackmem_malloc
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.4   __stack_chk_fail
0000000000000000      D  *UND*	0000000000000000              slurm_seterrno
0000000000000000      DF *UND*	0000000000000000              munge_encode
0000000000000000      DF *UND*	0000000000000000              munge_ctx_set
0000000000000000      D  *UND*	0000000000000000              slurm_xsignal
0000000000000000      DF *UND*	0000000000000000              munge_ctx_get
0000000000000000  w   D  *UND*	0000000000000000              __gmon_start__
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.2.5 strtol
0000000000000000      DF *UND*	0000000000000000              munge_ctx_strerror
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.2.5 time
0000000000000000      DF *UND*	0000000000000000              munge_ctx_create
0000000000000000      D  *UND*	0000000000000000              slurm_ctime2_r
0000000000000000      D  *UND*	0000000000000000              slurm_log_var
0000000000000000      D  *UND*	0000000000000000              slurm_get_ip_str
0000000000000000      D  *UND*	0000000000000000              get_name_info
0000000000000000      D  *UND*	0000000000000000              slurm_xcalloc
0000000000000000  w   D  *UND*	0000000000000000              _ITM_registerTMCloneTable
0000000000000000      D  *UND*	0000000000000000              slurm_auth_opts_to_socket
0000000000000000      D  *UND*	0000000000000000              slurm_conf
0000000000000000      D  *UND*	0000000000000000              slurm_packmem
0000000000000000      D  *UND*	0000000000000000              slurm_xfree
0000000000000000  w   DF *UND*	0000000000000000  GLIBC_2.2.5 __cxa_finalize
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.2.5 usleep
0000000000000000      D  *UND*	0000000000000000              slurm_error
0000000000002128 g    DO .rodata	0000000000000004  Base        plugin_version
0000000000001b92 g    DF .text	000000000000010e  Base        slurm_auth_get_host
0000000000001b42 g    DF .text	0000000000000028  Base        slurm_auth_get_uid
00000000000018ea g    DF .text	0000000000000258  Base        slurm_auth_verify
0000000000203104 g    D  .bss	0000000000000000  Base        __bss_start
0000000000001b6a g    DF .text	0000000000000028  Base        slurm_auth_get_gid
0000000000001ca0 g    DF .text	0000000000000084  Base        slurm_auth_pack
0000000000203108 g    D  .bss	0000000000000000  Base        _end
000000000000212c g    DO .rodata	0000000000000004  Base        plugin_id
000000000000160b g    DF .text	000000000000029b  Base        slurm_auth_create
0000000000001e21 g    DF .text	000000000000000a  Base        slurm_auth_thread_config
00000000000018a6 g    DF .text	0000000000000044  Base        slurm_auth_destroy
0000000000203104 g    D  .data	0000000000000000  Base        _edata
0000000000001590 g    DF .text	000000000000007b  Base        init
0000000000001e2b g    DF .text	0000000000000005  Base        slurm_auth_thread_clear
0000000000002130 g    DO .rodata	000000000000000b  Base        plugin_type
0000000000002140 g    DO .rodata	000000000000001c  Base        plugin_name
0000000000001d24 g    DF .text	00000000000000fd  Base        slurm_auth_unpack
0000000000001e30 g    DF .text	000000000000000a  Base        slurm_auth_token_generate

> > objdump -t /usr/lib64/slurm/auth_munge.so

# objdump -t /usr/lib64/slurm/auth_munge.so

/usr/lib64/slurm/auth_munge.so:     file format elf64-x86-64

SYMBOL TABLE:
0000000000000200 l    d  .note.gnu.build-id	0000000000000000              .note.gnu.build-id
0000000000000228 l    d  .gnu.hash	0000000000000000              .gnu.hash
00000000000002d8 l    d  .dynsym	0000000000000000              .dynsym
00000000000007b8 l    d  .dynstr	0000000000000000              .dynstr
0000000000000b30 l    d  .gnu.version	0000000000000000              .gnu.version
0000000000000b98 l    d  .gnu.version_r	0000000000000000              .gnu.version_r
0000000000000bc8 l    d  .rela.dyn	0000000000000000              .rela.dyn
0000000000000cb8 l    d  .rela.plt	0000000000000000              .rela.plt
0000000000000f70 l    d  .init	0000000000000000              .init
0000000000000f90 l    d  .plt	0000000000000000              .plt
0000000000001170 l    d  .plt.sec	0000000000000000              .plt.sec
0000000000001340 l    d  .text	0000000000000000              .text
0000000000001e3c l    d  .fini	0000000000000000              .fini
0000000000001e50 l    d  .rodata	0000000000000000              .rodata
000000000000215c l    d  .eh_frame_hdr	0000000000000000              .eh_frame_hdr
00000000000021e0 l    d  .eh_frame	0000000000000000              .eh_frame
00000000000023e8 l    d  .note.gnu.property	0000000000000000              .note.gnu.property
0000000000202d90 l    d  .init_array	0000000000000000              .init_array
0000000000202d98 l    d  .fini_array	0000000000000000              .fini_array
0000000000202da0 l    d  .data.rel.ro	0000000000000000              .data.rel.ro
0000000000202da8 l    d  .dynamic	0000000000000000              .dynamic
0000000000202fc8 l    d  .got	0000000000000000              .got
0000000000203000 l    d  .got.plt	0000000000000000              .got.plt
0000000000203100 l    d  .data	0000000000000000              .data
0000000000203104 l    d  .bss	0000000000000000              .bss
0000000000000000 l    d  .comment	0000000000000000              .comment
0000000000603108 l    d  .gnu.build.attributes	0000000000000000              .gnu.build.attributes
0000000000000000 l    d  .debug_aranges	0000000000000000              .debug_aranges
0000000000000000 l    d  .debug_info	0000000000000000              .debug_info
0000000000000000 l    d  .debug_abbrev	0000000000000000              .debug_abbrev
0000000000000000 l    d  .debug_line	0000000000000000              .debug_line
0000000000000000 l    d  .debug_str	0000000000000000              .debug_str
0000000000000000 l    d  .debug_loc	0000000000000000              .debug_loc
0000000000000000 l    d  .debug_ranges	0000000000000000              .debug_ranges
0000000000000000 l    d  .debug_macro	0000000000000000              .debug_macro
0000000000000000 l    df *ABS*	0000000000000000              auth_munge.c
00000000000013f9 l       .text	0000000000000000              .hidden .annobin_auth_munge.c
0000000000001e3a l       .text	0000000000000000              .hidden .annobin_auth_munge.c_end
0000000000001340 l       .text	0000000000000000              .hidden .annobin_auth_munge.c.hot
0000000000001340 l       .text	0000000000000000              .hidden .annobin_auth_munge.c_end.hot
0000000000001340 l       .text	0000000000000000              .hidden .annobin_auth_munge.c.unlikely
0000000000001340 l       .text	0000000000000000              .hidden .annobin_auth_munge.c_end.unlikely
0000000000001340 l       .text	0000000000000000              .hidden .annobin_auth_munge.c.startup
0000000000001340 l       .text	0000000000000000              .hidden .annobin_auth_munge.c_end.startup
0000000000001340 l       .text	0000000000000000              .hidden .annobin_auth_munge.c.exit
0000000000001340 l       .text	0000000000000000              .hidden .annobin_auth_munge.c_end.exit
00000000000013f9 l       .text	0000000000000000              .hidden .annobin__print_cred.start
00000000000013f9 l     F .text	0000000000000197              _print_cred
00000000000020e8 l     O .rodata	000000000000000c              __func__.17484
0000000000001590 l       .text	0000000000000000              .hidden .annobin__print_cred.end
0000000000001590 l       .text	0000000000000000              .hidden .annobin_init.start
0000000000203100 l     O .data	0000000000000004              bad_cred_test
0000000000002122 l     O .rodata	0000000000000005              __func__.17415
000000000000160b l       .text	0000000000000000              .hidden .annobin_init.end
000000000000160b l       .text	0000000000000000              .hidden .annobin_slurm_auth_create.start
0000000000002110 l     O .rodata	0000000000000012              __func__.17427
00000000000018a6 l       .text	0000000000000000              .hidden .annobin_slurm_auth_create.end
00000000000018a6 l       .text	0000000000000000              .hidden .annobin_slurm_auth_destroy.start
00000000000018ea l       .text	0000000000000000              .hidden .annobin_slurm_auth_destroy.end
00000000000018ea l       .text	0000000000000000              .hidden .annobin_slurm_auth_verify.start
00000000000020f8 l     O .rodata	000000000000000d              __func__.17475
0000000000001b42 l       .text	0000000000000000              .hidden .annobin_slurm_auth_verify.end
0000000000001b42 l       .text	0000000000000000              .hidden .annobin_slurm_auth_get_uid.start
0000000000001b6a l       .text	0000000000000000              .hidden .annobin_slurm_auth_get_uid.end
0000000000001b6a l       .text	0000000000000000              .hidden .annobin_slurm_auth_get_gid.start
0000000000001b92 l       .text	0000000000000000              .hidden .annobin_slurm_auth_get_gid.end
0000000000001b92 l       .text	0000000000000000              .hidden .annobin_slurm_auth_get_host.start
00000000000020d0 l     O .rodata	0000000000000014              __func__.17451
0000000000001ca0 l       .text	0000000000000000              .hidden .annobin_slurm_auth_get_host.end
0000000000001ca0 l       .text	0000000000000000              .hidden .annobin_slurm_auth_pack.start
00000000000020c0 l     O .rodata	0000000000000010              __func__.17458
0000000000001d24 l       .text	0000000000000000              .hidden .annobin_slurm_auth_pack.end
0000000000001d24 l       .text	0000000000000000              .hidden .annobin_slurm_auth_unpack.start
00000000000020a0 l     O .rodata	0000000000000012              __func__.17465
0000000000001e21 l       .text	0000000000000000              .hidden .annobin_slurm_auth_unpack.end
0000000000001e21 l       .text	0000000000000000              .hidden .annobin_slurm_auth_thread_config.start
0000000000001e2b l       .text	0000000000000000              .hidden .annobin_slurm_auth_thread_config.end
0000000000001e2b l       .text	0000000000000000              .hidden .annobin_slurm_auth_thread_clear.start
0000000000001e30 l       .text	0000000000000000              .hidden .annobin_slurm_auth_thread_clear.end
0000000000001e30 l       .text	0000000000000000              .hidden .annobin_slurm_auth_token_generate.start
0000000000001e3a l       .text	0000000000000000              .hidden .annobin_slurm_auth_token_generate.end
0000000000000000 l    df *ABS*	0000000000000000              crtstuff.c
0000000000001340 l     F .text	0000000000000000              deregister_tm_clones
0000000000001370 l     F .text	0000000000000000              register_tm_clones
00000000000013b0 l     F .text	0000000000000000              __do_global_dtors_aux
0000000000203104 l     O .bss	0000000000000001              completed.7294
0000000000202d98 l     O .fini_array	0000000000000000              __do_global_dtors_aux_fini_array_entry
00000000000013f0 l     F .text	0000000000000000              frame_dummy
0000000000202d90 l     O .init_array	0000000000000000              __frame_dummy_init_array_entry
0000000000000000 l    df *ABS*	0000000000000000              crtstuff.c
00000000000023e0 l     O .eh_frame	0000000000000000              __FRAME_END__
0000000000000000 l    df *ABS*	0000000000000000              
0000000000001e3c l     F .fini	0000000000000000              _fini
0000000000202da0 l     O .data.rel.ro	0000000000000000              __dso_handle
0000000000202da8 l     O .dynamic	0000000000000000              _DYNAMIC
000000000000215c l       .eh_frame_hdr	0000000000000000              __GNU_EH_FRAME_HDR
0000000000203108 l     O .data	0000000000000000              __TMC_END__
0000000000203000 l     O .got.plt	0000000000000000              _GLOBAL_OFFSET_TABLE_
0000000000000f70 l     F .init	0000000000000000              _init
0000000000000000       F *UND*	0000000000000000              getenv@@GLIBC_2.2.5
0000000000000000       F *UND*	0000000000000000              munge_decode
0000000000000000         *UND*	0000000000000000              slurm_get_log_level
0000000000000000       F *UND*	0000000000000000              free@@GLIBC_2.2.5
0000000000000000       F *UND*	0000000000000000              munge_ctx_destroy
0000000000000000         *UND*	0000000000000000              slurm_get_auth_ttl
0000000000000000  w      *UND*	0000000000000000              _ITM_deregisterTMCloneTable
0000000000000000         *UND*	0000000000000000              slurm_unpackmem_malloc
0000000000001b6a g     F .text	0000000000000028              slurm_auth_get_gid
0000000000203104 g       .data	0000000000000000              _edata
0000000000002130 g     O .rodata	000000000000000b              plugin_type
0000000000000000       F *UND*	0000000000000000              __stack_chk_fail@@GLIBC_2.4
0000000000000000         *UND*	0000000000000000              slurm_seterrno
0000000000000000       F *UND*	0000000000000000              munge_encode
0000000000000000       F *UND*	0000000000000000              munge_ctx_set
0000000000000000         *UND*	0000000000000000              slurm_xsignal
0000000000000000       F *UND*	0000000000000000              munge_ctx_get
0000000000000000  w      *UND*	0000000000000000              __gmon_start__
0000000000001e21 g     F .text	000000000000000a              slurm_auth_thread_config
0000000000000000       F *UND*	0000000000000000              strtol@@GLIBC_2.2.5
0000000000001ca0 g     F .text	0000000000000084              slurm_auth_pack
0000000000000000       F *UND*	0000000000000000              munge_ctx_strerror
0000000000001590 g     F .text	000000000000007b              init
0000000000000000       F *UND*	0000000000000000              time@@GLIBC_2.2.5
0000000000002140 g     O .rodata	000000000000001c              plugin_name
0000000000002128 g     O .rodata	0000000000000004              plugin_version
0000000000000000       F *UND*	0000000000000000              munge_ctx_create
0000000000000000         *UND*	0000000000000000              slurm_ctime2_r
0000000000000000         *UND*	0000000000000000              slurm_log_var
0000000000000000         *UND*	0000000000000000              slurm_get_ip_str
0000000000203108 g       .bss	0000000000000000              _end
00000000000018a6 g     F .text	0000000000000044              slurm_auth_destroy
0000000000001e2b g     F .text	0000000000000005              slurm_auth_thread_clear
0000000000000000         *UND*	0000000000000000              get_name_info
0000000000001e30 g     F .text	000000000000000a              slurm_auth_token_generate
0000000000203104 g       .bss	0000000000000000              __bss_start
0000000000001d24 g     F .text	00000000000000fd              slurm_auth_unpack
000000000000212c g     O .rodata	0000000000000004              plugin_id
000000000000160b g     F .text	000000000000029b              slurm_auth_create
0000000000000000         *UND*	0000000000000000              slurm_xcalloc
0000000000000000  w      *UND*	0000000000000000              _ITM_registerTMCloneTable
0000000000000000         *UND*	0000000000000000              slurm_auth_opts_to_socket
0000000000000000         *UND*	0000000000000000              slurm_conf
0000000000001b92 g     F .text	000000000000010e              slurm_auth_get_host
0000000000000000         *UND*	0000000000000000              slurm_packmem
0000000000000000         *UND*	0000000000000000              slurm_xfree
0000000000000000  w    F *UND*	0000000000000000              __cxa_finalize@@GLIBC_2.2.5
0000000000001b42 g     F .text	0000000000000028              slurm_auth_get_uid
0000000000000000       F *UND*	0000000000000000              usleep@@GLIBC_2.2.5
0000000000000000         *UND*	0000000000000000              slurm_error
00000000000018ea g     F .text	0000000000000258              slurm_auth_verify

> 
> > nm /usr/lib64/libslurm.so #path to libslurm.so may need to be corrected

Attached

> > ldd /usr/lib64/libslurm.so

# ldd /usr/lib64/libslurm.so
	linux-vdso.so.1 (0x00007fffd0d86000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f09c6aef000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f09c68cf000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f09c654d000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f09c6336000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f09c5f74000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f09c70bb000)

> > objdump -T /usr/lib64/libslurm.so

Attached

> > objdump -t /usr/lib64/libslurm.so

Attached

Thanks,

Torkil

Comment 7 Torkil Svensgaard 2020-11-13 00:09:56 MST

Created attachment 16620 [details]
objdump -T

Comment 8 Torkil Svensgaard 2020-11-13 00:10:21 MST

Created attachment 16621 [details]
objdump -t

Comment 9 Torkil Svensgaard 2020-11-13 00:10:42 MST

Created attachment 16622 [details]
nm libslurm

Comment 10 Nate Rini 2020-11-13 08:21:26 MST

Please also provide:
> lsb_release -a
> uname -a

Comment 11 Torkil Svensgaard 2020-11-13 08:30:02 MST

(In reply to Nate Rini from comment #10)
> Please also provide:
> > lsb_release -a

# lsb_release -a
LSB Version:	:core-4.1-amd64:core-4.1-noarch
Distributor ID:	CentOS
Description:	CentOS Linux release 8.2.2004 (Core) 
Release:	8.2.2004
Codename:	Core
# slurm/root ~ 

> > uname -a

# uname -a
Linux slurm.drcmr 4.18.0-193.28.1.el8_2.x86_64 #1 SMP Thu Oct 22 00:20:22 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Have a nice weekend,

Torkil

Comment 12 Nate Rini 2020-11-13 09:15:25 MST

Please also call this:
> ldd $(which scontrol)

Comment 13 Torkil Svensgaard 2020-11-13 09:40:26 MST

(In reply to Nate Rini from comment #12)
> Please also call this:
> > ldd $(which scontrol)

# ldd $(which scontrol)
	linux-vdso.so.1 (0x00007ffdf449b000)
	libslurmfull.so => /usr/local/lib/slurm/libslurmfull.so (0x00007fd34916a000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007fd348f66000)
	libm.so.6 => /lib64/libm.so.6 (0x00007fd348be4000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fd3489cd000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fd3487ad000)
	libc.so.6 => /lib64/libc.so.6 (0x00007fd3483eb000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fd349567000)
# slurm/root ~

Comment 14 Nate Rini 2020-11-13 09:47:04 MST

Please call:
> nm /usr/local/lib/slurm/libslurmfull.so

Comment 15 Torkil Svensgaard 2020-11-13 10:22:19 MST

Created attachment 16640 [details]
nm slurmfull.so

Comment 16 Nate Rini 2020-11-13 10:50:55 MST

(In reply to Torkil Svensgaard from comment #15)
> Created attachment 16640 [details]
> nm slurmfull.so

The installed slurmfull.so is missing the slurm_conf variable but has the old name of slurmctld_conf present. Looks like this file was not upgraded from 20.02.

Please call:
> rpm -ql slurm|grep libslurm
> rpm -Vav slurm

Comment 17 Torkil Svensgaard 2020-11-13 10:55:48 MST

(In reply to Nate Rini from comment #16)
> (In reply to Torkil Svensgaard from comment #15)
> > Created attachment 16640 [details]
> > nm slurmfull.so
> 
> The installed slurmfull.so is missing the slurm_conf variable but has the
> old name of slurmctld_conf present. Looks like this file was not upgraded
> from 20.02.
> 
> Please call:
> > rpm -ql slurm|grep libslurm

# rpm -ql slurm|grep libslurm
/usr/lib64/libslurm.so
/usr/lib64/libslurm.so.36
/usr/lib64/libslurm.so.36.0.0
/usr/lib64/slurm/libslurm_pmi.so
/usr/lib64/slurm/libslurmfull.so
# slurm/root ~ 

> > rpm -Vav slurm

# rpm -Vav slurm
.........    /usr/bin/sacct
.........    /usr/bin/sacctmgr
.........    /usr/bin/salloc
.........    /usr/bin/sattach
.........    /usr/bin/sbatch
.........    /usr/bin/sbcast
.........    /usr/bin/scancel
.........    /usr/bin/scontrol
.........    /usr/bin/scrontab
.........    /usr/bin/sdiag
.........    /usr/bin/sgather
.........    /usr/bin/sinfo
.........    /usr/bin/sprio
.........    /usr/bin/squeue
.........    /usr/bin/sreport
.........    /usr/bin/srun
.........    /usr/bin/sshare
.........    /usr/bin/sstat
.........    /usr/bin/strigger
.........  a /usr/lib/.build-id
.........  a /usr/lib/.build-id/04
.........  a /usr/lib/.build-id/04/881cc6bfa6e877bfcfcb12a9a2b51c78db04e1
.........  a /usr/lib/.build-id/06
.........  a /usr/lib/.build-id/06/69aebb17441ca751e776418d87a7b65d1eb96e
.........  a /usr/lib/.build-id/0a
.........  a /usr/lib/.build-id/0a/39acf651a42ca706de6403d0521a0ffe0256e1
.........  a /usr/lib/.build-id/0d
.........  a /usr/lib/.build-id/0d/362d1dbf25020e8b69f1e1beee0416cec31f06
.........  a /usr/lib/.build-id/0d/fd4db303d68589141c62608acaedde68eb67c0
.........  a /usr/lib/.build-id/0e
.........  a /usr/lib/.build-id/0e/48203bdcaa0008e3a8fb1bd92f198176326395
.........  a /usr/lib/.build-id/0f
.........  a /usr/lib/.build-id/0f/544f64bab851abe9155291995f8e764ca16e29
.........  a /usr/lib/.build-id/10
.........  a /usr/lib/.build-id/10/d860e692033ba746f35776ffe673230c67df90
.........  a /usr/lib/.build-id/13
.........  a /usr/lib/.build-id/13/8d492270744fa030d11726b9b368f0cda57abe
.........  a /usr/lib/.build-id/18
.........  a /usr/lib/.build-id/18/bd429447a33cf29d6cdf2eb4da43bec4fcfcf3
.........  a /usr/lib/.build-id/22
.........  a /usr/lib/.build-id/22/7f1076012c0b8fff518a879e87ae71d0296a48
.........  a /usr/lib/.build-id/25
.........  a /usr/lib/.build-id/25/fd5be676f813f926e5b8d920768d00e10a5cac
.........  a /usr/lib/.build-id/27
.........  a /usr/lib/.build-id/27/e879b02a811ea1853cd16d82f8eca9dfa9a114
.........  a /usr/lib/.build-id/28
.........  a /usr/lib/.build-id/28/2ce316aa82915e3c0b05b8cda3fe0316630232
.........  a /usr/lib/.build-id/29
.........  a /usr/lib/.build-id/29/f909c405cd53b66a0fc200e1d9791e85679ba2
.........  a /usr/lib/.build-id/2b
.........  a /usr/lib/.build-id/2b/515fa5fce7f1b6d9e5ed9aefa197dc13288976
.........  a /usr/lib/.build-id/2d
.........  a /usr/lib/.build-id/2d/195d29679bf2e5c1642c68f48ddd307fbb98f9
.........  a /usr/lib/.build-id/2d/d556b9ef3c5657059a1de3e910d57e816ed9c8
.........  a /usr/lib/.build-id/2e
.........  a /usr/lib/.build-id/2e/2caa1007f29f1091013ff3a80c6869dcd850d5
.........  a /usr/lib/.build-id/2f
.........  a /usr/lib/.build-id/2f/2d8911aa1454e772c36ad5ea664087e5176041
.........  a /usr/lib/.build-id/34
.........  a /usr/lib/.build-id/34/1e6bec32d477f43daa867bfe241f1d034bd48f
.........  a /usr/lib/.build-id/36
.........  a /usr/lib/.build-id/36/05c6659b7f088eebd2acb1c3c1d2873450c3f6
.........  a /usr/lib/.build-id/39
.........  a /usr/lib/.build-id/39/e08ee0c74f0cf4ccb3bc92391568f79750bbed
.........  a /usr/lib/.build-id/3b
.........  a /usr/lib/.build-id/3b/62605a060efa04eed12600bc6d2fd8953f78ff
.........  a /usr/lib/.build-id/3e
.........  a /usr/lib/.build-id/3e/b5d5c93f955cb51ad0dc72531674854fbd88cf
.........  a /usr/lib/.build-id/3e/e7bf731a3fa1b09065189e8b37aa8c048be561
.........  a /usr/lib/.build-id/3e/ff49b81a64e5804a5d23af9a4d5f67a4762ffa
.........  a /usr/lib/.build-id/3f
.........  a /usr/lib/.build-id/3f/efaef6861122962a0066fbcdcb22358271ceec
.........  a /usr/lib/.build-id/41
.........  a /usr/lib/.build-id/41/a8f83c8b521008a1d2f1a67de8a6d546a5ff64
.........  a /usr/lib/.build-id/42
.........  a /usr/lib/.build-id/42/a0ef0b86cdd23cc2f54e309ba0946321d6f36d
.........  a /usr/lib/.build-id/42/b083f72e34c0427f429f337e21c4520784ffe2
.........  a /usr/lib/.build-id/45
.........  a /usr/lib/.build-id/45/6960ac44d65cd95694d580b4303b20f6b7f521
.........  a /usr/lib/.build-id/45/dba9a984fd6f524d1bb84e54d133db9f663a73
.........  a /usr/lib/.build-id/49
.........  a /usr/lib/.build-id/49/57fd0ee880ab68ca0fc7954c8eab878745bde0
.........  a /usr/lib/.build-id/4a
.........  a /usr/lib/.build-id/4a/3c0f87928137839a9f847707ec1ecfbf431a15
.........  a /usr/lib/.build-id/4a/597709579ca9d0cdd238fae9dd2346d93c4fec
.........  a /usr/lib/.build-id/4f
.........  a /usr/lib/.build-id/4f/8aa4c0af488ff4c81f035908132eb894e8c7dd
.........  a /usr/lib/.build-id/52
.........  a /usr/lib/.build-id/52/4ce0d796d9b345533ae1859acac2709c9a99fd
.........  a /usr/lib/.build-id/57
.........  a /usr/lib/.build-id/57/4bdb83c4030526da11d43740231fdb182c3f64
.........  a /usr/lib/.build-id/5a
.........  a /usr/lib/.build-id/5a/a3e517b1ebc1854a4f5500aa4bf6b56d25aa30
.........  a /usr/lib/.build-id/5c
.........  a /usr/lib/.build-id/5c/d087df3413e8fa4815c392f871ab652022114e
.........  a /usr/lib/.build-id/61
.........  a /usr/lib/.build-id/61/16f8d2c3e4d4badd5af2b913eea9990f1bd6e1
.........  a /usr/lib/.build-id/61/e96dbc262cfd1fe04137cecff9f91c5d4dd856
.........  a /usr/lib/.build-id/62
.........  a /usr/lib/.build-id/62/742b6a83ff1b7dd5969f9e5aef31cb49ea426a
.........  a /usr/lib/.build-id/64
.........  a /usr/lib/.build-id/64/ea018c61453f890e4f5ffde30562e3b7eac6a2
.........  a /usr/lib/.build-id/67
.........  a /usr/lib/.build-id/67/81fb781dea8e8bfce0769f5b977e2e01f85ddb
.........  a /usr/lib/.build-id/68
.........  a /usr/lib/.build-id/68/0acb79ac50e46afa4442692bec32ede226d337
.........  a /usr/lib/.build-id/68/d28fccf152300da3263d6ea6afcceeb28d400c
.........  a /usr/lib/.build-id/6a
.........  a /usr/lib/.build-id/6a/f9e0d798c0c714376eb4fa46498cf6bdc9160b
.........  a /usr/lib/.build-id/6b
.........  a /usr/lib/.build-id/6b/1fb5dddeec6812eb5a3470d0f8ee26e543f51a
.........  a /usr/lib/.build-id/6c
.........  a /usr/lib/.build-id/6c/7e5212845e7ad3d850129f949174d2039d10cb
.........  a /usr/lib/.build-id/6e
.........  a /usr/lib/.build-id/6e/76a2293c914ea348d8ea4164451a1745f5384f
.........  a /usr/lib/.build-id/6e/eecc967fa774d0ec260769915e561b126e7078
.........  a /usr/lib/.build-id/70
.........  a /usr/lib/.build-id/70/bb0248c029faea1189ebe7465e7ce1675ce6f8
.........  a /usr/lib/.build-id/71
.........  a /usr/lib/.build-id/71/4632ff0a11ee055fcea882c824829eeaa9cbea
.........  a /usr/lib/.build-id/73
.........  a /usr/lib/.build-id/73/28f741c2c5d58ad9869664083f597c81ca88c1
.........  a /usr/lib/.build-id/74
.........  a /usr/lib/.build-id/74/b8ed5615b3fe80405e089bdd00d4509a272a44
.........  a /usr/lib/.build-id/75
.........  a /usr/lib/.build-id/75/0bcee0e465f44debea6fdf8a6bf7f26066126a
.........  a /usr/lib/.build-id/77
.........  a /usr/lib/.build-id/77/da419d4a92f2270de5042188640f9c2c782bbd
.........  a /usr/lib/.build-id/7b
.........  a /usr/lib/.build-id/7b/200d50ae66a00f7156eee163c1fc8356ab8b6e
.........  a /usr/lib/.build-id/80
.........  a /usr/lib/.build-id/80/96a890f3b4d7a2dca98d32dc5f0124c50df47f
.........  a /usr/lib/.build-id/82
.........  a /usr/lib/.build-id/82/7b5f22feb90a4428de3412fffdce40df16de60
.........  a /usr/lib/.build-id/85
.........  a /usr/lib/.build-id/85/d316ea949f83f2c5c6c303b60ee5c8887f61cb
.........  a /usr/lib/.build-id/86
.........  a /usr/lib/.build-id/86/5e126d6e8d1c5113f657fb039e9ca02c836e0e
.........  a /usr/lib/.build-id/87
.........  a /usr/lib/.build-id/87/0fc805cd8fb483499a2baea2dd66799410b315
.........  a /usr/lib/.build-id/87/28ca287ae3f4e5cbe0e06eea8fb7a98bbdecb2
.........  a /usr/lib/.build-id/88
.........  a /usr/lib/.build-id/88/42ab167e0ac1fe6e5858e91b076437c1aece04
.........  a /usr/lib/.build-id/8b
.........  a /usr/lib/.build-id/8b/be631578c426b43a54e87978b5ff8c00d14c2c
.........  a /usr/lib/.build-id/90
.........  a /usr/lib/.build-id/90/88d8286e8be13bbd511cbaf2306018c174a7e7
.........  a /usr/lib/.build-id/95
.........  a /usr/lib/.build-id/95/30a32a09cdd13718955d7398c147b1451804cd
.........  a /usr/lib/.build-id/96
.........  a /usr/lib/.build-id/96/0e494db7a2efd26f8a3620d44056cc8297bcde
.........  a /usr/lib/.build-id/9a
.........  a /usr/lib/.build-id/9a/7357d86310fef3597db9c77aad50f8b77556d6
.........  a /usr/lib/.build-id/9a/fb899f399fd6302e3814aa8fecd904202d8b58
.........  a /usr/lib/.build-id/9e
.........  a /usr/lib/.build-id/9e/c6af910edf8c150abe88698b1b3948918ea5a3
.........  a /usr/lib/.build-id/a2
.........  a /usr/lib/.build-id/a2/b924d9b0e9f1e8b64d18e741873a6d8f71e072
.........  a /usr/lib/.build-id/a5
.........  a /usr/lib/.build-id/a5/add0f6ef876774be8cd760626f891e347605cc
.........  a /usr/lib/.build-id/a6
.........  a /usr/lib/.build-id/a6/8ca0bcc66f460933f4ba6545144577df13bca8
.........  a /usr/lib/.build-id/a8
.........  a /usr/lib/.build-id/a8/5d6eebcf52d1cd81717c9f3643bd0d94edad64
.........  a /usr/lib/.build-id/ac
.........  a /usr/lib/.build-id/ac/78c581eaa5e101b17c299d9ead211a4a18ed5c
.........  a /usr/lib/.build-id/b1
.........  a /usr/lib/.build-id/b1/24c0047377e0dfef7e1d31571a1b7f6f20f50c
.........  a /usr/lib/.build-id/b1/7c69b1bb75b823ee9fc78b0199cfb38606c3b7
.........  a /usr/lib/.build-id/b6
.........  a /usr/lib/.build-id/b6/615623fff0a70ad2b7b97406ea33b8930e6ab2
.........  a /usr/lib/.build-id/b6/6eef086e008cb8071ea7e0af84402bfa3c2159
.........  a /usr/lib/.build-id/bb
.........  a /usr/lib/.build-id/bb/c94c701a25a459cd12453684e2cfb1cb7bcfd8
.........  a /usr/lib/.build-id/bc
.........  a /usr/lib/.build-id/bc/9b6ba4e48031d0f2a83004bedd36b361ff8a7d
.........  a /usr/lib/.build-id/bd
.........  a /usr/lib/.build-id/bd/6c157e41023b38ca79ddfa759027db261fbc40
.........  a /usr/lib/.build-id/bd/b47c4ef88cb8b19119ac774f4a4febf775dfa1
.........  a /usr/lib/.build-id/c4
.........  a /usr/lib/.build-id/c4/bcf14c598a5c84d84257b127342e08bfe818b8
.........  a /usr/lib/.build-id/c6
.........  a /usr/lib/.build-id/c6/bdb17dbf7b72997f503f121f0ff8e953fcbd40
.........  a /usr/lib/.build-id/cc
.........  a /usr/lib/.build-id/cc/c56ca2b8e2e00415996e1868396aed3ffaf8c7
.........  a /usr/lib/.build-id/d3
.........  a /usr/lib/.build-id/d3/0de5bddcf40106d4f34a74ea927c8ed2862cf4
.........  a /usr/lib/.build-id/d3/3e05f050903b93eaaac67eb9646f777b8d4689
.........  a /usr/lib/.build-id/d5
.........  a /usr/lib/.build-id/d5/3e4a882bf6f52c01310b8c955f745d14ac7e51
.........  a /usr/lib/.build-id/d6
.........  a /usr/lib/.build-id/d6/9a59cf43b618e92bb4e7fe08cb78f642c95701
.........  a /usr/lib/.build-id/d8
.........  a /usr/lib/.build-id/d8/0c48d1a4f975841dce32a304e512e4da2f81fb
.........  a /usr/lib/.build-id/d8/68b915578b1b819a0dfa04a8265b685128c877
.........  a /usr/lib/.build-id/d9
.........  a /usr/lib/.build-id/d9/bffebefc41b80536feb95656425e9ee1beca27
.........  a /usr/lib/.build-id/e3
.........  a /usr/lib/.build-id/e3/0bac0b08d12299b70fa43d85fb6f4dfbdebbd0
.........  a /usr/lib/.build-id/e4
.........  a /usr/lib/.build-id/e4/eeeb76f14d39e5f8d0419fb73e533a0915b027
.........  a /usr/lib/.build-id/ec
.........  a /usr/lib/.build-id/ec/9172555f7528da00e828e4e6dedb253be24fec
.........  a /usr/lib/.build-id/ed
.........  a /usr/lib/.build-id/ed/adf5c2695e3f82016f03c56b4021c0507fadc9
.........  a /usr/lib/.build-id/f0
.........  a /usr/lib/.build-id/f0/072606042ec44a486b991d6c790c5e2b79cd6c
.........  a /usr/lib/.build-id/f0/e1adeab5ba4e6cdf7ce698f3339c56b786c40b
.........  a /usr/lib/.build-id/f1
.........  a /usr/lib/.build-id/f1/b2d07971574ece282c1518103f1fd16cddfebf
.........  a /usr/lib/.build-id/f3
.........  a /usr/lib/.build-id/f3/435e03e8359ffbedc7d0e54e1065b2d7adbf29
.........  a /usr/lib/.build-id/fa
.........  a /usr/lib/.build-id/fa/8b3e793926a4cea3e3dce28819d90caf050728
.........  a /usr/lib/.build-id/fa/ce6ba6d1dbf94235975d4d2042e154cf76d86f
.........  a /usr/lib/.build-id/ff
.........  a /usr/lib/.build-id/ff/c6819de2cc7de9ea4684983e9fe736fe89b532
.........    /usr/lib64/libnss_slurm.so.2
.........    /usr/lib64/libslurm.so
.........    /usr/lib64/libslurm.so.36
.........    /usr/lib64/libslurm.so.36.0.0
.........    /usr/lib64/slurm/accounting_storage_none.so
.........    /usr/lib64/slurm/accounting_storage_slurmdbd.so
.........    /usr/lib64/slurm/acct_gather_energy_ibmaem.so
.........    /usr/lib64/slurm/acct_gather_energy_none.so
.........    /usr/lib64/slurm/acct_gather_energy_pm_counters.so
.........    /usr/lib64/slurm/acct_gather_energy_rapl.so
.........    /usr/lib64/slurm/acct_gather_filesystem_lustre.so
.........    /usr/lib64/slurm/acct_gather_filesystem_none.so
.........    /usr/lib64/slurm/acct_gather_interconnect_none.so
.........    /usr/lib64/slurm/acct_gather_profile_none.so
.........    /usr/lib64/slurm/auth_munge.so
.........    /usr/lib64/slurm/burst_buffer_datawarp.so
.........    /usr/lib64/slurm/burst_buffer_generic.so
.........    /usr/lib64/slurm/cli_filter_none.so
.........    /usr/lib64/slurm/cli_filter_syslog.so
.........    /usr/lib64/slurm/cli_filter_user_defaults.so
.........    /usr/lib64/slurm/core_spec_cray_aries.so
.........    /usr/lib64/slurm/core_spec_none.so
.........    /usr/lib64/slurm/cred_munge.so
.........    /usr/lib64/slurm/cred_none.so
.........    /usr/lib64/slurm/ext_sensors_none.so
.........    /usr/lib64/slurm/gpu_generic.so
.........    /usr/lib64/slurm/gres_gpu.so
.........    /usr/lib64/slurm/gres_mic.so
.........    /usr/lib64/slurm/gres_mps.so
.........    /usr/lib64/slurm/gres_nic.so
.........    /usr/lib64/slurm/job_container_cncu.so
.........    /usr/lib64/slurm/job_container_none.so
.........    /usr/lib64/slurm/job_submit_all_partitions.so
.........    /usr/lib64/slurm/job_submit_cray_aries.so
.........    /usr/lib64/slurm/job_submit_require_timelimit.so
.........    /usr/lib64/slurm/job_submit_throttle.so
.........    /usr/lib64/slurm/jobacct_gather_cgroup.so
.........    /usr/lib64/slurm/jobacct_gather_linux.so
.........    /usr/lib64/slurm/jobacct_gather_none.so
.........    /usr/lib64/slurm/jobcomp_filetxt.so
.........    /usr/lib64/slurm/jobcomp_mysql.so
.........    /usr/lib64/slurm/jobcomp_none.so
.........    /usr/lib64/slurm/jobcomp_script.so
.........    /usr/lib64/slurm/launch_slurm.so
.........    /usr/lib64/slurm/libslurm_pmi.so
.........    /usr/lib64/slurm/libslurmfull.so
.........    /usr/lib64/slurm/mcs_account.so
.........    /usr/lib64/slurm/mcs_group.so
.........    /usr/lib64/slurm/mcs_none.so
.........    /usr/lib64/slurm/mcs_user.so
.........    /usr/lib64/slurm/mpi_cray_shasta.so
.........    /usr/lib64/slurm/mpi_none.so
.........    /usr/lib64/slurm/mpi_pmi2.so
.........    /usr/lib64/slurm/node_features_knl_cray.so
.........    /usr/lib64/slurm/node_features_knl_generic.so
.........    /usr/lib64/slurm/power_cray_aries.so
.........    /usr/lib64/slurm/power_none.so
.........    /usr/lib64/slurm/preempt_none.so
.........    /usr/lib64/slurm/preempt_partition_prio.so
.........    /usr/lib64/slurm/preempt_qos.so
.........    /usr/lib64/slurm/prep_script.so
.........    /usr/lib64/slurm/priority_basic.so
.........    /usr/lib64/slurm/priority_multifactor.so
.........    /usr/lib64/slurm/proctrack_cgroup.so
.........    /usr/lib64/slurm/proctrack_linuxproc.so
.........    /usr/lib64/slurm/proctrack_pgid.so
.........    /usr/lib64/slurm/route_default.so
.........    /usr/lib64/slurm/route_topology.so
.........    /usr/lib64/slurm/sched_backfill.so
.........    /usr/lib64/slurm/sched_builtin.so
.........    /usr/lib64/slurm/sched_hold.so
.........    /usr/lib64/slurm/select_cons_res.so
.........    /usr/lib64/slurm/select_cons_tres.so
.........    /usr/lib64/slurm/select_cray_aries.so
.........    /usr/lib64/slurm/select_linear.so
.........    /usr/lib64/slurm/site_factor_none.so
.........    /usr/lib64/slurm/slurmctld_nonstop.so
.........    /usr/lib64/slurm/src
.........    /usr/lib64/slurm/src/sattach
.........    /usr/lib64/slurm/src/sattach/sattach.wrapper.c
.........    /usr/lib64/slurm/src/srun
.........    /usr/lib64/slurm/src/srun/srun.wrapper.c
.........    /usr/lib64/slurm/switch_cray_aries.so
.........    /usr/lib64/slurm/switch_none.so
.........    /usr/lib64/slurm/task_affinity.so
.........    /usr/lib64/slurm/task_cgroup.so
.........    /usr/lib64/slurm/task_cray_aries.so
.........    /usr/lib64/slurm/task_none.so
.........    /usr/lib64/slurm/topology_3d_torus.so
.........    /usr/lib64/slurm/topology_hypercube.so
.........    /usr/lib64/slurm/topology_none.so
.........    /usr/lib64/slurm/topology_tree.so
.........    /usr/share/doc
.........    /usr/share/doc/slurm-20.11.0-0rc1
.........    /usr/share/doc/slurm-20.11.0-0rc1/html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/Slurm_Entity.pdf
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/Slurm_Individual.pdf
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/accounting.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/accounting_storageplugins.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/acct_gather_energy_plugins.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/acct_gather_profile_plugins.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/add.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/allocation_pies.gif
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/api.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/arch.gif
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/authplugins.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/bb_plugins.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/big_sys.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/burst_buffer.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/cgroups.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/classic_fair_share.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/cli_filter_plugins.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/coding_style.pdf
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/configless_slurm.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/configurator.easy.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/configurator.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/cons_res.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/cons_res_share.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/containers.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/contributor.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/core_spec.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/core_spec_plugins.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/cpu_management.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/cray.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/cred_plugins.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/disclaimer.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/dist_plane.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/documentation.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/download.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/elastic_computing.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/elasticsearch.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/entities.gif
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/example_usage.gif
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/ext_sensorsplugins.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/fair_tree.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/faq.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/federation.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/fonts.css
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/fonts.ttf
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/gang_scheduling.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/gres.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/gres_design.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/gres_plugins.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/hdf5_job_outline.png
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/hdf5_profile_user_guide.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/hdf5_task_attr.png
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/heterogeneous_jobs.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/high_throughput.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/ibm_pe_fig1.png
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/ibm_pe_fig2.png
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/intel_knl.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/job_array.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/job_container_plugins.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/job_exit_code.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/job_launch.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/job_submit_plugins.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/jobacct_gatherplugins.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/jobcompplugins.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/jquery.min.js
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/jwt.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/k_function.gif
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/launch_plugins.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/licenses.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/mail.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/man_index.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/mc_support.gif
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/mc_support.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/mcs.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/mcs_plugins.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/meetings.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/mpi_guide.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/mpiplugins.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/multi_cluster.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/network.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/network_failover.gif
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/network_federation.gif
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/network_multi_cluster.gif
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/network_srun.gif
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/network_standard.gif
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/news.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/node_features_plugins.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/nss_slurm.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/overview.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/pam_slurm_adopt.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/plane_ex1.gif
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/plane_ex2.gif
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/plane_ex3.gif
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/plane_ex4.gif
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/plane_ex5.gif
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/plane_ex6.gif
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/plane_ex7.gif
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/platforms.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/plugins.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/power_mgmt.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/power_plugins.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/power_save.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/preempt.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/preemption_plugins.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/prep_plugins.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/priority_multifactor.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/priority_multifactor3.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/priority_plugins.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/proctrack_plugins.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/programmer_guide.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/prolog_epilog.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/publications.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/qos.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/quickstart.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/quickstart_admin.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/reservations.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/reset.css
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/resource_binding.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/resource_limits.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/rest.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/rosetta.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/route_plugin.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/sched_config.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/schedmd.png
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/schedplugins.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/select_design.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/selectplugins.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/site_factor.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/slurm.css
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/slurm.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/slurm_logo.png
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/slurm_ug_agenda.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/slurmctld_plugstack.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/squeue_color.png
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/style.css
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/switchplugins.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/taskplugins.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/team.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/testimonials.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/topo_ex1.gif
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/topo_ex2.gif
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/topology.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/topology_plugin.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/tres.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/troubleshoot.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/tutorial_intro_files.tar
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/tutorials.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/usage_pies.gif
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/user_permissions.html
.........  d /usr/share/doc/slurm-20.11.0-0rc1/html/wckey.html
.........    /usr/share/man
.........    /usr/share/man/man1
.........  d /usr/share/man/man1/sacct.1.gz
.........  d /usr/share/man/man1/sacctmgr.1.gz
.........  d /usr/share/man/man1/salloc.1.gz
.........  d /usr/share/man/man1/sattach.1.gz
.........  d /usr/share/man/man1/sbatch.1.gz
.........  d /usr/share/man/man1/sbcast.1.gz
.........  d /usr/share/man/man1/scancel.1.gz
.........  d /usr/share/man/man1/scontrol.1.gz
.........  d /usr/share/man/man1/scrontab.1.gz
.........  d /usr/share/man/man1/sdiag.1.gz
.........  d /usr/share/man/man1/sgather.1.gz
.........  d /usr/share/man/man1/sinfo.1.gz
.........  d /usr/share/man/man1/slurm.1.gz
.........  d /usr/share/man/man1/sprio.1.gz
.........  d /usr/share/man/man1/squeue.1.gz
.........  d /usr/share/man/man1/sreport.1.gz
.........  d /usr/share/man/man1/srun.1.gz
.........  d /usr/share/man/man1/sshare.1.gz
.........  d /usr/share/man/man1/sstat.1.gz
.........  d /usr/share/man/man1/strigger.1.gz
.........    /usr/share/man/man3
.........  d /usr/share/man/man3/Slurm.3pm.gz
.........  d /usr/share/man/man3/Slurm::Bitstr.3pm.gz
.........  d /usr/share/man/man3/Slurm::Constant.3pm.gz
.........  d /usr/share/man/man3/Slurm::Hostlist.3pm.gz
.........  d /usr/share/man/man3/Slurm::Stepctx.3pm.gz
.........  d /usr/share/man/man3/Slurmdb.3pm.gz
.........  d /usr/share/man/man3/slurm_allocate_het_job_blocking.3.gz
.........  d /usr/share/man/man3/slurm_allocate_resources.3.gz
.........  d /usr/share/man/man3/slurm_allocate_resources_blocking.3.gz
.........  d /usr/share/man/man3/slurm_allocation_lookup.3.gz
.........  d /usr/share/man/man3/slurm_allocation_msg_thr_create.3.gz
.........  d /usr/share/man/man3/slurm_allocation_msg_thr_destroy.3.gz
.........  d /usr/share/man/man3/slurm_api_version.3.gz
.........  d /usr/share/man/man3/slurm_clear_trigger.3.gz
.........  d /usr/share/man/man3/slurm_complete_job.3.gz
.........  d /usr/share/man/man3/slurm_confirm_allocation.3.gz
.........  d /usr/share/man/man3/slurm_create_partition.3.gz
.........  d /usr/share/man/man3/slurm_create_reservation.3.gz
.........  d /usr/share/man/man3/slurm_delete_partition.3.gz
.........  d /usr/share/man/man3/slurm_delete_reservation.3.gz
.........  d /usr/share/man/man3/slurm_free_ctl_conf.3.gz
.........  d /usr/share/man/man3/slurm_free_front_end_info_msg.3.gz
.........  d /usr/share/man/man3/slurm_free_job_alloc_info_response_msg.3.gz
.........  d /usr/share/man/man3/slurm_free_job_array_resp.3.gz
.........  d /usr/share/man/man3/slurm_free_job_info_msg.3.gz
.........  d /usr/share/man/man3/slurm_free_job_step_create_response_msg.3.gz
.........  d /usr/share/man/man3/slurm_free_job_step_info_response_msg.3.gz
.........  d /usr/share/man/man3/slurm_free_node_info.3.gz
.........  d /usr/share/man/man3/slurm_free_node_info_msg.3.gz
.........  d /usr/share/man/man3/slurm_free_partition_info.3.gz
.........  d /usr/share/man/man3/slurm_free_partition_info_msg.3.gz
.........  d /usr/share/man/man3/slurm_free_reservation_info_msg.3.gz
.........  d /usr/share/man/man3/slurm_free_resource_allocation_response_msg.3.gz
.........  d /usr/share/man/man3/slurm_free_slurmd_status.3.gz
.........  d /usr/share/man/man3/slurm_free_submit_response_response_msg.3.gz
.........  d /usr/share/man/man3/slurm_free_trigger_msg.3.gz
.........  d /usr/share/man/man3/slurm_get_end_time.3.gz
.........  d /usr/share/man/man3/slurm_get_errno.3.gz
.........  d /usr/share/man/man3/slurm_get_job_steps.3.gz
.........  d /usr/share/man/man3/slurm_get_rem_time.3.gz
.........  d /usr/share/man/man3/slurm_get_triggers.3.gz
.........  d /usr/share/man/man3/slurm_het_job_lookup.3.gz
.........  d /usr/share/man/man3/slurm_het_job_will_run.3.gz
.........  d /usr/share/man/man3/slurm_hostlist_create.3.gz
.........  d /usr/share/man/man3/slurm_hostlist_destroy.3.gz
.........  d /usr/share/man/man3/slurm_hostlist_shift.3.gz
.........  d /usr/share/man/man3/slurm_init_job_desc_msg.3.gz
.........  d /usr/share/man/man3/slurm_init_part_desc_msg.3.gz
.........  d /usr/share/man/man3/slurm_init_resv_desc_msg.3.gz
.........  d /usr/share/man/man3/slurm_init_trigger_msg.3.gz
.........  d /usr/share/man/man3/slurm_init_update_front_end_msg.3.gz
.........  d /usr/share/man/man3/slurm_init_update_node_msg.3.gz
.........  d /usr/share/man/man3/slurm_init_update_step_msg.3.gz
.........  d /usr/share/man/man3/slurm_job_cpus_allocated_on_node.3.gz
.........  d /usr/share/man/man3/slurm_job_cpus_allocated_on_node_id.3.gz
.........  d /usr/share/man/man3/slurm_job_step_create.3.gz
.........  d /usr/share/man/man3/slurm_job_step_launch_t_init.3.gz
.........  d /usr/share/man/man3/slurm_job_step_layout_free.3.gz
.........  d /usr/share/man/man3/slurm_job_step_layout_get.3.gz
.........  d /usr/share/man/man3/slurm_job_will_run.3.gz
.........  d /usr/share/man/man3/slurm_job_will_run2.3.gz
.........  d /usr/share/man/man3/slurm_jobinfo_ctx_get.3.gz
.........  d /usr/share/man/man3/slurm_kill_job.3.gz
.........  d /usr/share/man/man3/slurm_kill_job_step.3.gz
.........  d /usr/share/man/man3/slurm_load_ctl_conf.3.gz
.........  d /usr/share/man/man3/slurm_load_front_end.3.gz
.........  d /usr/share/man/man3/slurm_load_job.3.gz
.........  d /usr/share/man/man3/slurm_load_job_user.3.gz
.........  d /usr/share/man/man3/slurm_load_jobs.3.gz
.........  d /usr/share/man/man3/slurm_load_node.3.gz
.........  d /usr/share/man/man3/slurm_load_node_single.3.gz
.........  d /usr/share/man/man3/slurm_load_partitions.3.gz
.........  d /usr/share/man/man3/slurm_load_reservations.3.gz
.........  d /usr/share/man/man3/slurm_load_slurmd_status.3.gz
.........  d /usr/share/man/man3/slurm_notify_job.3.gz
.........  d /usr/share/man/man3/slurm_perror.3.gz
.........  d /usr/share/man/man3/slurm_pid2jobid.3.gz
.........  d /usr/share/man/man3/slurm_ping.3.gz
.........  d /usr/share/man/man3/slurm_print_ctl_conf.3.gz
.........  d /usr/share/man/man3/slurm_print_front_end_info_msg.3.gz
.........  d /usr/share/man/man3/slurm_print_front_end_table.3.gz
.........  d /usr/share/man/man3/slurm_print_job_info.3.gz
.........  d /usr/share/man/man3/slurm_print_job_info_msg.3.gz
.........  d /usr/share/man/man3/slurm_print_job_step_info.3.gz
.........  d /usr/share/man/man3/slurm_print_job_step_info_msg.3.gz
.........  d /usr/share/man/man3/slurm_print_node_info_msg.3.gz
.........  d /usr/share/man/man3/slurm_print_node_table.3.gz
.........  d /usr/share/man/man3/slurm_print_partition_info.3.gz
.........  d /usr/share/man/man3/slurm_print_partition_info_msg.3.gz
.........  d /usr/share/man/man3/slurm_print_reservation_info.3.gz
.........  d /usr/share/man/man3/slurm_print_reservation_info_msg.3.gz
.........  d /usr/share/man/man3/slurm_print_slurmd_status.3.gz
.........  d /usr/share/man/man3/slurm_read_hostfile.3.gz
.........  d /usr/share/man/man3/slurm_reconfigure.3.gz
.........  d /usr/share/man/man3/slurm_requeue.3.gz
.........  d /usr/share/man/man3/slurm_requeue2.3.gz
.........  d /usr/share/man/man3/slurm_resume.3.gz
.........  d /usr/share/man/man3/slurm_resume2.3.gz
.........  d /usr/share/man/man3/slurm_set_debug_level.3.gz
.........  d /usr/share/man/man3/slurm_set_trigger.3.gz
.........  d /usr/share/man/man3/slurm_shutdown.3.gz
.........  d /usr/share/man/man3/slurm_signal_job.3.gz
.........  d /usr/share/man/man3/slurm_signal_job_step.3.gz
.........  d /usr/share/man/man3/slurm_slurmd_status.3.gz
.........  d /usr/share/man/man3/slurm_sprint_front_end_table.3.gz
.........  d /usr/share/man/man3/slurm_sprint_job_info.3.gz
.........  d /usr/share/man/man3/slurm_sprint_job_step_info.3.gz
.........  d /usr/share/man/man3/slurm_sprint_node_table.3.gz
.........  d /usr/share/man/man3/slurm_sprint_partition_info.3.gz
.........  d /usr/share/man/man3/slurm_sprint_reservation_info.3.gz
.........  d /usr/share/man/man3/slurm_step_ctx_create.3.gz
.........  d /usr/share/man/man3/slurm_step_ctx_create_no_alloc.3.gz
.........  d /usr/share/man/man3/slurm_step_ctx_daemon_per_node_hack.3.gz
.........  d /usr/share/man/man3/slurm_step_ctx_destroy.3.gz
.........  d /usr/share/man/man3/slurm_step_ctx_get.3.gz
.........  d /usr/share/man/man3/slurm_step_ctx_params_t_init.3.gz
.........  d /usr/share/man/man3/slurm_step_launch.3.gz
.........  d /usr/share/man/man3/slurm_step_launch_abort.3.gz
.........  d /usr/share/man/man3/slurm_step_launch_fwd_signal.3.gz
.........  d /usr/share/man/man3/slurm_step_launch_wait_finish.3.gz
.........  d /usr/share/man/man3/slurm_step_launch_wait_start.3.gz
.........  d /usr/share/man/man3/slurm_strerror.3.gz
.........  d /usr/share/man/man3/slurm_submit_batch_job.3.gz
.........  d /usr/share/man/man3/slurm_suspend.3.gz
.........  d /usr/share/man/man3/slurm_suspend2.3.gz
.........  d /usr/share/man/man3/slurm_takeover.3.gz
.........  d /usr/share/man/man3/slurm_terminate_job.3.gz
.........  d /usr/share/man/man3/slurm_terminate_job_step.3.gz
.........  d /usr/share/man/man3/slurm_update_front_end.3.gz
.........  d /usr/share/man/man3/slurm_update_job.3.gz
.........  d /usr/share/man/man3/slurm_update_job2.3.gz
.........  d /usr/share/man/man3/slurm_update_node.3.gz
.........  d /usr/share/man/man3/slurm_update_partition.3.gz
.........  d /usr/share/man/man3/slurm_update_reservation.3.gz
.........  d /usr/share/man/man3/slurm_update_step.3.gz
.........    /usr/share/man/man5
.........  d /usr/share/man/man5/acct_gather.conf.5.gz
.........  d /usr/share/man/man5/burst_buffer.conf.5.gz
.........  d /usr/share/man/man5/cgroup.conf.5.gz
.........  d /usr/share/man/man5/ext_sensors.conf.5.gz
.........  d /usr/share/man/man5/gres.conf.5.gz
.........  d /usr/share/man/man5/knl.conf.5.gz
.........  d /usr/share/man/man5/nonstop.conf.5.gz
.........  d /usr/share/man/man5/slurm.conf.5.gz
.........  d /usr/share/man/man5/slurmdbd.conf.5.gz
.........  d /usr/share/man/man5/topology.conf.5.gz
.........    /usr/share/man/man8
.........  d /usr/share/man/man8/slurmctld.8.gz
.........  d /usr/share/man/man8/slurmd.8.gz
.........  d /usr/share/man/man8/slurmdbd.8.gz
.........  d /usr/share/man/man8/slurmrestd.8.gz
.........  d /usr/share/man/man8/slurmstepd.8.gz
.........  d /usr/share/man/man8/spank.8.gz

Comment 18 Nate Rini 2020-11-13 11:00:59 MST

(In reply to Torkil Svensgaard from comment #17)
> (In reply to Nate Rini from comment #16)
> > (In reply to Torkil Svensgaard from comment #15)
> # rpm -Vav slurm
> .........    /usr/lib64/slurm/libslurmfull.so

I would expect this result if the file was not matching the rpm:
> S.5....T.    /usr/lib64/slurm/libslurmfull.so

How did you build the RPMs?

Comment 19 Torkil Svensgaard 2020-11-13 11:06:50 MST

(In reply to Nate Rini from comment #18)
> (In reply to Torkil Svensgaard from comment #17)
> > (In reply to Nate Rini from comment #16)
> > > (In reply to Torkil Svensgaard from comment #15)
> > # rpm -Vav slurm
> > .........    /usr/lib64/slurm/libslurmfull.so
> 
> I would expect this result if the file was not matching the rpm:
> > S.5....T.    /usr/lib64/slurm/libslurmfull.so
> 
> How did you build the RPMs?

Last run I deleted the old rpmbuild directory and then ran as root:

"
rpmbuild -ta slurm-20.11.0-0rc1.tar.bz2 --with mysql
"

Then installed all the rpms in /root/rpmbuild/RPMS/x86_64/

Mvh.

Torkil

Comment 20 Nate Rini 2020-11-13 15:11:59 MST

(In reply to Torkil Svensgaard from comment #19)
> rpmbuild -ta slurm-20.11.0-0rc1.tar.bz2 --with mysql

My attempts to replicate using your same distro and package have failed. We will need to see what is happening on your build.

Please try calling this instead:
> rpmbuild -D 'noclean 1'  -D 'rel 1' -ta slurm-20.11.0-0rc1.tar.bz2 --with mysql

Once it is complete, please zip and attach:
/root/rpmbuild/BUILD/slurm-20.11.0-0rc1/config.log

Comment 21 Torkil Svensgaard 2020-11-14 00:56:32 MST

Created attachment 16659 [details]
config.log

Comment 22 Nate Rini 2020-11-16 10:43:25 MST

Please call this and attach (please correct path if needed):
> nm /root/rpmbuild/BUILD/slurm-20.11.0-0rc1/src/api/libslurmfull.so

Comment 23 Torkil Svensgaard 2020-11-17 00:32:07 MST

(In reply to Nate Rini from comment #22)
> Please call this and attach (please correct path if needed):
> > nm /root/rpmbuild/BUILD/slurm-20.11.0-0rc1/src/api/libslurmfull.so

Hmm, not sure if this is the one you want:

"
# slurm/root rpmbuild 
# find . -name libslurmfull*
./BUILD/slurm-20.11.0-0rc1/src/api/.libs/libslurmfull.so
./BUILD/slurm-20.11.0-0rc1/src/api/.libs/libslurmfull.a
./BUILD/slurm-20.11.0-0rc1/src/api/.libs/libslurmfull.lai
./BUILD/slurm-20.11.0-0rc1/src/api/.libs/libslurmfull.la
./BUILD/slurm-20.11.0-0rc1/src/api/libslurmfull.la

# nm ./BUILD/slurm-20.11.0-0rc1/src/api/.libs/libslurmfull.so > /tmp/libslurmful.so.nm
"

Attaching.

Comment 24 Torkil Svensgaard 2020-11-17 00:32:57 MST

Created attachment 16681 [details]
nm libslurmfull.so

Comment 26 Nate Rini 2020-11-17 13:01:52 MST

(In reply to Torkil Svensgaard from comment #23)
> (In reply to Nate Rini from comment #22)
> > Please call this and attach (please correct path if needed):
> > > nm /root/rpmbuild/BUILD/slurm-20.11.0-0rc1/src/api/libslurmfull.so
> # nm ./BUILD/slurm-20.11.0-0rc1/src/api/.libs/libslurmfull.so >
> /tmp/libslurmful.so.nm

(In reply to Torkil Svensgaard from comment #24)
> Created attachment 16681 [details]
> nm libslurmfull.so
Yes, that is the correct one. With this output, we can confirm the compiled library has the correct symbol, so somehow RPM is not installing or it is getting overwritten or mislinked.

Let's make sure the 2 files are the same:
> sha1sum /root/rpmbuild/BUILD/slurm-20.11.0-0rc1/src/api/libslurmfull.so
> sha1sum /usr/lib64/slurm/libslurmfull.so

Comment 27 Nate Rini 2020-11-17 13:02:33 MST

Also, if possible, please swap to 20.11.0 as there have been a good number of bug fixes since rc1 (although none related directly to this ticket).

Comment 28 Torkil Svensgaard 2020-11-17 13:35:39 MST

(In reply to Nate Rini from comment #26)
> (In reply to Torkil Svensgaard from comment #23)
> > (In reply to Nate Rini from comment #22)
> > > Please call this and attach (please correct path if needed):
> > > > nm /root/rpmbuild/BUILD/slurm-20.11.0-0rc1/src/api/libslurmfull.so
> > # nm ./BUILD/slurm-20.11.0-0rc1/src/api/.libs/libslurmfull.so >
> > /tmp/libslurmful.so.nm
> 
> (In reply to Torkil Svensgaard from comment #24)
> > Created attachment 16681 [details]
> > nm libslurmfull.so
> Yes, that is the correct one. With this output, we can confirm the compiled
> library has the correct symbol, so somehow RPM is not installing or it is
> getting overwritten or mislinked.
> 
> Let's make sure the 2 files are the same:
> > sha1sum /root/rpmbuild/BUILD/slurm-20.11.0-0rc1/src/api/libslurmfull.so
> > sha1sum /usr/lib64/slurm/libslurmfull.so

# sha1sum ./rpmbuild/BUILD/slurm-20.11.0-0rc1/src/api/.libs/libslurmfull.so
14a29ae27f2d8fc98f155d4d7e1e50dde69e2143  ./rpmbuild/BUILD/slurm-20.11.0-0rc1/src/api/.libs/libslurmfull.so
# slurm/root ~ 
# sha1sum /usr/lib64/slurm/libslurmfull.so
80cad7d14f7a810e136405a46db18d6d54b8500c  /usr/lib64/slurm/libslurmfull.so

Note the .libs part missing from what you pasted. Not sure if that significant.

Comment 29 Torkil Svensgaard 2020-11-17 13:44:55 MST

(In reply to Nate Rini from comment #27)
> Also, if possible, please swap to 20.11.0 as there have been a good number
> of bug fixes since rc1 (although none related directly to this ticket).

Tried that, same issue with lingering 20.02.5:

"
# dnf install slurm-perlapi-20.11.0-1.el8.x86_64.rpm slurm-20.11.0-1.el8.x86_64.rpm 
Last metadata expiration check: 0:16:21 ago on Tue 17 Nov 2020 09:27:14 PM CET.
Error: 
 Problem: cannot install both slurm-20.11.0-1.el8.x86_64 and slurm-20.02.5-1.el8.x86_64
  - package slurm-perlapi-20.11.0-1.el8.x86_64 requires libslurm.so.35()(64bit), but none of the providers can be installed
  - conflicting requests
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
# slurm/root x86_64 
"

Comment 30 Nate Rini 2020-11-17 13:59:16 MST

(In reply to Torkil Svensgaard from comment #28)
> # sha1sum ./rpmbuild/BUILD/slurm-20.11.0-0rc1/src/api/.libs/libslurmfull.so
> 14a29ae27f2d8fc98f155d4d7e1e50dde69e2143 
> ./rpmbuild/BUILD/slurm-20.11.0-0rc1/src/api/.libs/libslurmfull.so
>
> # sha1sum /usr/lib64/slurm/libslurmfull.so
> 80cad7d14f7a810e136405a46db18d6d54b8500c  /usr/lib64/slurm/libslurmfull.so

This confirms the files are not the same. Can you please just overwrite `/usr/lib64/slurm/libslurmfull.so` with `./rpmbuild/BUILD/slurm-20.11.0-0rc1/src/api/.libs/libslurmfull.so` and see if it works?

> Note the .libs part missing from what you pasted. Not sure if that
> significant.

They should be the same but it wouldn't hurt to check the sha1sum on them too.

Comment 31 Torkil Svensgaard 2020-11-17 14:13:25 MST

(In reply to Nate Rini from comment #30)
> (In reply to Torkil Svensgaard from comment #28)
> > # sha1sum ./rpmbuild/BUILD/slurm-20.11.0-0rc1/src/api/.libs/libslurmfull.so
> > 14a29ae27f2d8fc98f155d4d7e1e50dde69e2143 
> > ./rpmbuild/BUILD/slurm-20.11.0-0rc1/src/api/.libs/libslurmfull.so
> >
> > # sha1sum /usr/lib64/slurm/libslurmfull.so
> > 80cad7d14f7a810e136405a46db18d6d54b8500c  /usr/lib64/slurm/libslurmfull.so
> 
> This confirms the files are not the same. Can you please just overwrite
> `/usr/lib64/slurm/libslurmfull.so` with
> `./rpmbuild/BUILD/slurm-20.11.0-0rc1/src/api/.libs/libslurmfull.so` and see
> if it works?

sinfo still fails
 
> > Note the .libs part missing from what you pasted. Not sure if that
> > significant.
> 
> They should be the same but it wouldn't hurt to check the sha1sum on them
> too.

I only had the one in the .lib dir.

Comment 32 Nate Rini 2020-11-17 14:24:56 MST

(In reply to Torkil Svensgaard from comment #31)
> sinfo still fails

Please call sinfo with these env variables:
> env LD_DEBUG_OUTPUT=sym.log LD_DEBUG=all sinfo

Please attach syum.log.

Comment 33 Torkil Svensgaard 2020-11-17 14:55:11 MST

Created attachment 16699 [details]
sym.log

Comment 34 Nate Rini 2020-11-17 20:29:56 MST

(In reply to Torkil Svensgaard from comment #33)
> Created attachment 16699 [details]
> sym.log

The log shows it loaded just fine while linking to libslurmfull.so.

Looks like we are going to have to call scontrol instead:
> env LD_DEBUG_OUTPUT=sym2.log LD_DEBUG=all scontrol reconfigure

Comment 35 Torkil Svensgaard 2020-11-19 00:07:50 MST

Created attachment 16732 [details]
Sym2.log

Comment 36 Torkil Svensgaard 2020-11-19 00:09:22 MST

(In reply to Nate Rini from comment #34)
> (In reply to Torkil Svensgaard from comment #33)
> > Created attachment 16699 [details]
> > sym.log
> 
> The log shows it loaded just fine while linking to libslurmfull.so.
> 
> Looks like we are going to have to call scontrol instead:
> > env LD_DEBUG_OUTPUT=sym2.log LD_DEBUG=all scontrol reconfigure

That command didn't didn't return either:

"
# env LD_DEBUG_OUTPUT=sym2.log LD_DEBUG=all scontrol reconfigure
scontrol: error: _fetch_child: failed to fetch remote configs
^C
"

Uploaded log after I stopped it. 

Mvh.

Torkil

Comment 37 Marcin Stolarek 2020-11-19 03:08:26 MST

Could you please attach slurmctld log?

Comment 38 Torkil Svensgaard 2020-11-19 03:37:21 MST

Created attachment 16734 [details]
slurmctld.log

Comment 39 Marcin Stolarek 2020-11-19 03:48:14 MST

The last lines in the log are:
>[2020-11-13T07:56:06.561] Terminate signal (SIGINT or SIGTERM) received
>[2020-11-13T07:56:06.624] Saving all slurm state

Is slurmctld running and writing to the file?

cheers,
Marcin

Comment 41 Nate Rini 2020-11-19 13:29:37 MST

(In reply to Torkil Svensgaard from comment #36)
> (In reply to Nate Rini from comment #34)
> > (In reply to Torkil Svensgaard from comment #33)
> > Looks like we are going to have to call scontrol instead:
> > > env LD_DEBUG_OUTPUT=sym2.log LD_DEBUG=all scontrol reconfigure
> 
> That command didn't didn't return either:

The log shows that scontrol is linking correctly. We are going to have to try this with slurmctld now. Looks like it is hanging as slurmctld is not coming back.

You will probably need to run this as the slurmuser or root:
> env LD_DEBUG_OUTPUT=sym3.log LD_DEBUG=all slurmctld -Dvv

Comment 42 Torkil Svensgaard 2020-11-19 23:33:07 MST

(In reply to Nate Rini from comment #41)
 
> You will probably need to run this as the slurmuser or root:
> > env LD_DEBUG_OUTPUT=sym3.log LD_DEBUG=all slurmctld -Dvv

Hmm, wrong version of slurmctld:

"
# env LD_DEBUG_OUTPUT=sym3.log LD_DEBUG=all slurmctld -Dvv -f /etc/slurm/slurm.conf
slurmctld: error: If using PrologFlags=Contain for pam_slurm_adopt, either proctrack/cgroup or proctrack/cray_aries is required.  If not using pam_slurm_adopt, please ignore error.
slurmctld: debug:  Log file re-opened
slurmctld: pidfile not locked, assuming no running daemon
slurmctld: error: Job accounting information gathered, but not stored
slurmctld: slurmctld version 20.02.5 started on cluster drcmr
slurmctld: error: Couldn't find the specified plugin name for cred/munge looking at all files
slurmctld: error: cannot find cred plugin for cred/munge
slurmctld: error: cannot create cred context for cred/munge
slurmctld: fatal: slurm_cred_creator_ctx_create((null)): Operation not permitted
# slurm/root tmp 
# env LD_DEBUG_OUTPUT=sym3_with_f.log LD_DEBUG=all slurmctld -Dvv -f /etc/slurm/slurm.conf
slurmctld: error: If using PrologFlags=Contain for pam_slurm_adopt, either proctrack/cgroup or proctrack/cray_aries is required.  If not using pam_slurm_adopt, please ignore error.
slurmctld: debug:  Log file re-opened
slurmctld: pidfile not locked, assuming no running daemon
slurmctld: error: Job accounting information gathered, but not stored
slurmctld: slurmctld version 20.02.5 started on cluster drcmr
slurmctld: error: Couldn't find the specified plugin name for cred/munge looking at all files
slurmctld: error: cannot find cred plugin for cred/munge
slurmctld: error: cannot create cred context for cred/munge
slurmctld: fatal: slurm_cred_creator_ctx_create((null)): Operation not permitted
# slurm/root tmp 
# env LD_DEBUG_OUTPUT=sym3.log LD_DEBUG=all slurmctld -Dvv 
slurmctld: error: s_p_parse_file: unable to status file /usr/local/etc/slurm.conf: No such file or directory, retrying in 1sec up to 60sec
slurmctld: error: ClusterName needs to be specified
slurmctld: fatal: Unable to process configuration file
"

Comment 43 Torkil Svensgaard 2020-11-19 23:34:08 MST

Created attachment 16752 [details]
sym3_with_f

Comment 44 Torkil Svensgaard 2020-11-19 23:34:30 MST

Created attachment 16753 [details]
sym3

Comment 45 Nate Rini 2020-11-20 08:05:47 MST

(In reply to Torkil Svensgaard from comment #42)
> (In reply to Nate Rini from comment #41)
> # env LD_DEBUG_OUTPUT=sym3.log LD_DEBUG=all slurmctld -Dvv -f
> slurmctld: slurmctld version 20.02.5 started on cluster drcmr

Does this node host both versions?

Can we try this?
> env LD_DEBUG_OUTPUT=sym3.log LD_DEBUG=all /usr/local/sbin/slurmctld -Dvv -f /etc/slurm/slurm.conf
> which slurmctld

How is the slurmctld daemon normally being started?

Comment 46 Torkil Svensgaard 2020-11-20 13:07:40 MST

Created attachment 16764 [details]
sym4.log - which slurmctld

Comment 47 Torkil Svensgaard 2020-11-20 13:08:46 MST

(In reply to Nate Rini from comment #45)
> (In reply to Torkil Svensgaard from comment #42)
> > (In reply to Nate Rini from comment #41)
> > # env LD_DEBUG_OUTPUT=sym3.log LD_DEBUG=all slurmctld -Dvv -f
> > slurmctld: slurmctld version 20.02.5 started on cluster drcmr
> 
> Does this node host both versions?

Not by design. I had 20.02.5 on it first and then tried to just upgrade the packages.

> Can we try this?
> > env LD_DEBUG_OUTPUT=sym3.log LD_DEBUG=all /usr/local/sbin/slurmctld -Dvv -f /etc/slurm/slurm.conf
> > which slurmctld

Uploaded as sym4.log

> How is the slurmctld daemon normally being started?

Systemd

Mvh.

Torkil

Comment 48 Torkil Svensgaard 2020-11-30 02:20:59 MST

Hi

Any suggestions, shourt of reinstalling the box? I would like to get SLURM up and running =)

Comment 49 Nate Rini 2020-11-30 10:13:09 MST

(In reply to Torkil Svensgaard from comment #48)
> Any suggestions, shourt of reinstalling the box? I would like to get SLURM
> up and running =)

While that is always an option: I'm currently working on suggestions.

Comment 51 Nate Rini 2020-11-30 11:53:31 MST

Please call this:
> mv /usr/local/lib/slurm/auth_munge.so /usr/local/lib/slurm/auth_munge.so.disabled
> mv /usr/lib64/slurm/auth_munge.so /usr/lib64/slurm/auth_munge.so.disabled

Then please re-install all of the Slurm RPMs.

Please attach the slurmctld log and the same debug log from comment #45 if it keeps failing.

Comment 52 Torkil Svensgaard 2020-11-30 12:04:45 MST

(In reply to Nate Rini from comment #51)
> Please call this:
> > mv /usr/local/lib/slurm/auth_munge.so /usr/local/lib/slurm/auth_munge.so.disabled

I actually deleted /usr/local/lib/slurm earlier today, after removing all slurm packages, as I thought it was strange it wasn't deleted by uninstalling

> > mv /usr/lib64/slurm/auth_munge.so /usr/lib64/slurm/auth_munge.so.disabled

Done.

> Then please re-install all of the Slurm RPMs.

Done, now getting this error:

"
# sinfo
sinfo: error while loading shared libraries: libslurmfull.so: cannot open shared object file: No such file or directory
" 

> Please attach the slurmctld log and the same debug log from comment #45 if
> it keeps failing.

Incoming shortly.

Mvh.

Torkil

Comment 53 Nate Rini 2020-11-30 12:07:14 MST

(In reply to Torkil Svensgaard from comment #52)
> # sinfo
> sinfo: error while loading shared libraries: libslurmfull.so: cannot open
> shared object file: No such file or directory

Please call
> ldd sinfo

Comment 54 Torkil Svensgaard 2020-11-30 12:09:07 MST

Created attachment 16867 [details]
sym3v2

Comment 55 Torkil Svensgaard 2020-11-30 12:11:18 MST

Created attachment 16869 [details]
slurmctld.log

Comment 56 Torkil Svensgaard 2020-11-30 12:13:39 MST

(In reply to Nate Rini from comment #53)
> (In reply to Torkil Svensgaard from comment #52)
> > # sinfo
> > sinfo: error while loading shared libraries: libslurmfull.so: cannot open
> > shared object file: No such file or directory
> 
> Please call
> > ldd sinfo

"
 ldd /usr/local/bin/sinfo
        linux-vdso.so.1 (0x00007ffd36ed7000)
        libslurmfull.so => not found
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f41ab864000)
        libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f41ab64d000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f41ab42d000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f41ab06b000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f41aba68000)
"

I've also installed munge from source earlier btw, which was slightly never that the one in the repos. I rebuilt everything afterwards.

Mvh.

Torkil

Comment 57 Nate Rini 2020-11-30 12:14:55 MST

(In reply to Torkil Svensgaard from comment #55)
> Created attachment 16869 [details]
> slurmctld.log

Did slurmctld load successfully?

Comment 58 Nate Rini 2020-11-30 12:16:46 MST

(In reply to Torkil Svensgaard from comment #56)
> I've also installed munge from source earlier btw, which was slightly never
> that the one in the repos. I rebuilt everything afterwards.

Let's just clear all libraries from Slurm:
> mv /usr/lib64/slurm /tmp/slurm_lib_removed

Comment 59 Torkil Svensgaard 2020-11-30 12:17:12 MST

(In reply to Nate Rini from comment #57)
> (In reply to Torkil Svensgaard from comment #55)
> > Created attachment 16869 [details]
> > slurmctld.log
> 
> Did slurmctld load successfully?

I believe so:

"
# slurm/root ~
# systemctl status slurmctld
● slurmctld.service - Slurm controller daemon
   Loaded: loaded (/usr/lib/systemd/system/slurmctld.service; disabled; vendor >
   Active: active (running) since Mon 2020-11-30 20:02:57 CET; 12min ago
 Main PID: 248821 (slurmctld)
    Tasks: 7
   Memory: 4.2M
   CGroup: /system.slice/slurmctld.service
           └─248821 /usr/sbin/slurmctld -D

Nov 30 20:02:57 slurm.drcmr systemd[1]: Stopped Slurm controller daemon.
Nov 30 20:02:57 slurm.drcmr systemd[1]: Started Slurm controller daemon.
"

Comment 60 Torkil Svensgaard 2020-11-30 12:18:01 MST

(In reply to Nate Rini from comment #58)
> (In reply to Torkil Svensgaard from comment #56)
> > I've also installed munge from source earlier btw, which was slightly never
> > that the one in the repos. I rebuilt everything afterwards.
> 
> Let's just clear all libraries from Slurm:
> > mv /usr/lib64/slurm /tmp/slurm_lib_removed

Done, then reinstall?

Comment 61 Nate Rini 2020-11-30 12:22:33 MST

(In reply to Torkil Svensgaard from comment #60)
> (In reply to Nate Rini from comment #58)
> > (In reply to Torkil Svensgaard from comment #56)
> > > I've also installed munge from source earlier btw, which was slightly never
> > > that the one in the repos. I rebuilt everything afterwards.
> > 
> > Let's just clear all libraries from Slurm:
> > > mv /usr/lib64/slurm /tmp/slurm_lib_removed
> 
> Done, then reinstall?

Yes, for reasons currently unclear there are files left in /usr/lib64/slurm even though all of them should be in /usr/lcoal/lib/slurm.

Comment 62 Torkil Svensgaard 2020-11-30 12:27:38 MST

(In reply to Nate Rini from comment #61)
> (In reply to Torkil Svensgaard from comment #60)
> > (In reply to Nate Rini from comment #58)
> > > (In reply to Torkil Svensgaard from comment #56)
> > > > I've also installed munge from source earlier btw, which was slightly never
> > > > that the one in the repos. I rebuilt everything afterwards.
> > > 
> > > Let's just clear all libraries from Slurm:
> > > > mv /usr/lib64/slurm /tmp/slurm_lib_removed
> > 
> > Done, then reinstall?
> 
> Yes, for reasons currently unclear there are files left in /usr/lib64/slurm
> even though all of them should be in /usr/lcoal/lib/slurm.

Re-install puts files in /usr/lib64/slurm again and still missing libslurmfull.so.

Move out of the way again and build fresh packages maybe? Or purge all slurm packages instead of just reinstall? 

Mvh.

Torkil

Comment 63 Torkil Svensgaard 2020-11-30 12:27:48 MST

(In reply to Nate Rini from comment #61)
> (In reply to Torkil Svensgaard from comment #60)
> > (In reply to Nate Rini from comment #58)
> > > (In reply to Torkil Svensgaard from comment #56)
> > > > I've also installed munge from source earlier btw, which was slightly never
> > > > that the one in the repos. I rebuilt everything afterwards.
> > > 
> > > Let's just clear all libraries from Slurm:
> > > > mv /usr/lib64/slurm /tmp/slurm_lib_removed
> > 
> > Done, then reinstall?
> 
> Yes, for reasons currently unclear there are files left in /usr/lib64/slurm
> even though all of them should be in /usr/lcoal/lib/slurm.

Re-install puts files in /usr/lib64/slurm again and still missing libslurmfull.so.

Move out of the way again and build fresh packages maybe? Or purge all slurm packages instead of just reinstall? 

Mvh.

Torkil

Comment 65 Nate Rini 2020-11-30 12:38:09 MST

(In reply to Torkil Svensgaard from comment #63)
> (In reply to Nate Rini from comment #61)
> > (In reply to Torkil Svensgaard from comment #60)
> > > (In reply to Nate Rini from comment #58)
> > > > (In reply to Torkil Svensgaard from comment #56)
> > > > > I've also installed munge from source earlier btw, which was slightly never
> > > > > that the one in the repos. I rebuilt everything afterwards.
> > > > 
> > > > Let's just clear all libraries from Slurm:
> > > > > mv /usr/lib64/slurm /tmp/slurm_lib_removed
> > > 
> > > Done, then reinstall?
> > 
> > Yes, for reasons currently unclear there are files left in /usr/lib64/slurm
> > even though all of them should be in /usr/lcoal/lib/slurm.
> 
> Re-install puts files in /usr/lib64/slurm again and still missing
> libslurmfull.so.

Looking at config.log from comment#21
>  --libdir=/usr/lib64
All the libraries are expected to be /usr/lib64/ and not /usr/local/.

Is it possible, if just for testing, to do the following:

> mv /usr/local /usr/local.disabled

Looks like the RPATH may be getting inherited from the /usr/local install.

Comment 66 Torkil Svensgaard 2020-11-30 12:41:45 MST

(In reply to Nate Rini from comment #65)
> (In reply to Torkil Svensgaard from comment #63)
> > (In reply to Nate Rini from comment #61)
> > > (In reply to Torkil Svensgaard from comment #60)
> > > > (In reply to Nate Rini from comment #58)
> > > > > (In reply to Torkil Svensgaard from comment #56)
> > > > > > I've also installed munge from source earlier btw, which was slightly never
> > > > > > that the one in the repos. I rebuilt everything afterwards.
> > > > > 
> > > > > Let's just clear all libraries from Slurm:
> > > > > > mv /usr/lib64/slurm /tmp/slurm_lib_removed
> > > > 
> > > > Done, then reinstall?
> > > 
> > > Yes, for reasons currently unclear there are files left in /usr/lib64/slurm
> > > even though all of them should be in /usr/lcoal/lib/slurm.
> > 
> > Re-install puts files in /usr/lib64/slurm again and still missing
> > libslurmfull.so.
> 
> Looking at config.log from comment#21
> >  --libdir=/usr/lib64
> All the libraries are expected to be /usr/lib64/ and not /usr/local/.
> 
> Is it possible, if just for testing, to do the following:
> 
> > mv /usr/local /usr/local.disabled

That seems to work!

> Looks like the RPATH may be getting inherited from the /usr/local install.

Ok. What's the fix for that? =)

Mvh.

Torkil

Comment 67 Nate Rini 2020-11-30 12:44:07 MST

(In reply to Torkil Svensgaard from comment #66)
> > > mv /usr/local /usr/local.disabled
> 
> That seems to work!
> 
> > Looks like the RPATH may be getting inherited from the /usr/local install.
> 
> Ok. What's the fix for that? =)

Looks like a previous install of Slurm was in /usr/local but never cleaned up. I suggest disabling (or purging) all of the slurm related files in /usr/local as the RPMs (as compiled) expect this in / instead.

Comment 68 Nate Rini 2020-11-30 12:44:33 MST

(In reply to Nate Rini from comment #67)
> (In reply to Torkil Svensgaard from comment #66)
> > > > mv /usr/local /usr/local.disabled
> > 
> > That seems to work!
> > 
> > > Looks like the RPATH may be getting inherited from the /usr/local install.
> > 
> > Ok. What's the fix for that? =)
> 
> Looks like a previous install of Slurm was in /usr/local but never cleaned
> up. I suggest disabling (or purging) all of the slurm related files in
> /usr/local as the RPMs (as compiled) expect this in / instead.

Or just remove /usr/local/* from your PATH

Comment 69 Torkil Svensgaard 2020-11-30 12:50:04 MST

(In reply to Nate Rini from comment #67)
> (In reply to Torkil Svensgaard from comment #66)
> > > > mv /usr/local /usr/local.disabled
> > 
> > That seems to work!
> > 
> > > Looks like the RPATH may be getting inherited from the /usr/local install.
> > 
> > Ok. What's the fix for that? =)
> 
> Looks like a previous install of Slurm was in /usr/local but never cleaned
> up. I suggest disabling (or purging) all of the slurm related files in
> /usr/local as the RPMs (as compiled) expect this in / instead.

I removed the files in /usr/local/bin, that's seems to do it.

Comment 70 Nate Rini 2020-11-30 12:54:07 MST

(In reply to Torkil Svensgaard from comment #69)
> I removed the files in /usr/local/bin, that's seems to do it.

I suggest considering a clean install in the long term but for now: it is working. I'm going to close this ticket. Please respond if you have any questions.

Thanks,
--Nate

Comment 71 Torkil Svensgaard 2020-11-30 12:55:02 MST

(In reply to Torkil Svensgaard from comment #69)
> (In reply to Nate Rini from comment #67)
> > (In reply to Torkil Svensgaard from comment #66)
> > > > > mv /usr/local /usr/local.disabled
> > > 
> > > That seems to work!
> > > 
> > > > Looks like the RPATH may be getting inherited from the /usr/local install.
> > > 
> > > Ok. What's the fix for that? =)
> > 
> > Looks like a previous install of Slurm was in /usr/local but never cleaned
> > up. I suggest disabling (or purging) all of the slurm related files in
> > /usr/local as the RPMs (as compiled) expect this in / instead.
> 
> I removed the files in /usr/local/bin, that's seems to do it.

I can't seem to resume bigger9 though, but that's probably not related?

From slurmctld.log:

"
[2020-11-30T20:52:50.104] error: _slurm_rpc_node_registration node=bigger9: Invalid argument
"

I haven't made configuration changes for 20.11 so something essential could have changed I guess

Comment 72 Torkil Svensgaard 2020-11-30 13:07:35 MST

All good, thanks.

Mvh.

Torkil