Ticket 5796

Hi David,

Can you please describe exactly how you did perform the upgrade?

Old nodes should still be able to work exactly as new nodes since they can talk to the controller.
What are the exact errors you are finding?

Your seff script may be the one from the old release of Slurm, so when running from the new nodes
the calls that it does may be looking for symbols that are not in the same file than before
(the library name has changed from libslurm to libslurmfull.so). seff uses the Perl API.

Can you take a look at your /local/software/slurm/bin/seff and see if it is the new or the old one?

Created attachment 7939 [details]
slurm-upgrade.txt

Hi Felip,


Thank you for your reply. First of all, I confirm that (as suspected) there are differences between my version of seff and that provided by slurm v18.08. On the other hand, I'm not convinced that there are major differences that could account for the issue that I reported. I have provided with you with the output from diff comparing the two scripts -- see below.


Furthermore, I confirm that the behaviour of the two scripts are identical. That is, they both give the following output...


[root@blue51 slurm]# /usr/bin/seff 151807
perl: error: plugin_load_from_file: dlopen(/usr/lib64/slurm/accounting_storage_slurmdbd.so): /usr/lib64/slurm/accounting_storage_slurmdbd.so: undefined symbol: node_record_count
perl: error: Couldn't load specified plugin name for accounting_storage/slurmdbd: Dlopen of plugin file failed
perl: error: cannot create accounting_storage context for accounting_storage/slurmdbd
perl: error: plugin_load_from_file: dlopen(/usr/lib64/slurm/accounting_storage_slurmdbd.so): /usr/lib64/slurm/accounting_storage_slurmdbd.so: undefined symbol: node_record_count
perl: error: Couldn't load specified plugin name for accounting_storage/slurmdbd: Dlopen of plugin file failed
perl: error: cannot create accounting_storage context for accounting_storage/slurmdbd

Please take a closer look at the output of the diff command and the output.


With regard to my slurm upgrade. I can confirm that I followed a procedure as suggested and approved by your colleagues. For the sake of the argument I have attached a file of the upgrade plan that was followed. The cluster is stable following the upgrade and slurm database commands respond as expected.


Best regards,

David


[root@blue51 slurm]# diff /usr/bin/seff /local/software/slurm/bin/seff
13a14,15
> use Term::ANSIColor;
> #$Term::ANSIColor::AUTORESET = 1;
50c52
< $job_cond{flags} = JOBCOND_FLAG_NO_TRUNC;
---
> $job_cond{without_usage_truncation} = 1;
168,170c170,180
<     printf("CPU Efficiency: %.2f%% of %s core-walltime\n", $cpu_eff, time2str($corewalltime));
<     printf("Job Wall-clock time: %s\n", time2str($walltime));
<     if ($ntasks == 1) {
---
>      if ($cpu_eff < 50) {
>         #printf color('bold red');
>         printf("**** Please note that the CPU efficency of your job is less than 50%% ****\n");
>         printf("**** Ask the HPC team for advice -- if you need it\n");
>         printf("**** CPU Efficiency: %.2f%% of %s core-walltime\n", $cpu_eff, time2str($corewalltime));
>         printf("\n");
>      } else {
>         printf("CPU Efficiency: %.2f%% of %s core-walltime\n", $cpu_eff, time2str($corewalltime));
>      }
>      #printf("CPU Efficiency: %.2f%% of %s core-walltime\n", $cpu_eff, time2str($corewalltime));
>      if ($ntasks == 1) {
178,180c188,190
<     } else {
<         $mem_eff = 0.0;
<     }
---
> #    } else {
> #        $mem_eff = 0.0;
> #    }
188a199
>     }



________________________________
From: bugs@schedmd.com <bugs@schedmd.com>
Sent: 01 October 2018 19:09
To: Baker D.J.
Subject: [Bug 5796] Missing accounting plugins in slurm 18.08 installation


Comment # 1<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.schedmd.com%2Fshow_bug.cgi%3Fid%3D5796%23c1&data=01%7C01%7Cd.j.baker%40soton.ac.uk%7Cb1cd83024d124cee550a08d627c904d2%7C4a5378f929f44d3ebe89669d03ada9d8%7C1&sdata=87Jj1j%2FdwL6VwEzBZDAuHyMAirj5PFT2iX3bZSWhkig%3D&reserved=0> on bug 5796<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.schedmd.com%2Fshow_bug.cgi%3Fid%3D5796&data=01%7C01%7Cd.j.baker%40soton.ac.uk%7Cb1cd83024d124cee550a08d627c904d2%7C4a5378f929f44d3ebe89669d03ada9d8%7C1&sdata=WBffNT8OfhwilW6kp5DG6JI0QDX8z7ATexlI0aajchc%3D&reserved=0> from Felip Moll<mailto:felip.moll@schedmd.com>

Hi David,

Can you please describe exactly how you did perform the upgrade?

Old nodes should still be able to work exactly as new nodes since they can talk
to the controller.
What are the exact errors you are finding?

Your seff script may be the one from the old release of Slurm, so when running
from the new nodes
the calls that it does may be looking for symbols that are not in the same file
than before
(the library name has changed from libslurm to libslurmfull.so). seff uses the
Perl API.

Can you take a look at your /local/software/slurm/bin/seff and see if it is the
new or the old one?

________________________________
You are receiving this mail because:

  *   You reported the bug.

Hi Felip,


This issue is not a show stopper, however I wondered if you had any chance to make any progress on the failure of the seff command. Please let me know as/when you have any updates.


Best regards,

David

________________________________
From: Baker D.J.
Sent: 03 October 2018 16:37
To: bugs@schedmd.com
Subject: Re: [Bug 5796] Missing accounting plugins in slurm 18.08 installation


Hi Felip,


Thank you for your reply. First of all, I confirm that (as suspected) there are differences between my version of seff and that provided by slurm v18.08. On the other hand, I'm not convinced that there are major differences that could account for the issue that I reported. I have provided with you with the output from diff comparing the two scripts -- see below.


Furthermore, I confirm that the behaviour of the two scripts are identical. That is, they both give the following output...


[root@blue51 slurm]# /usr/bin/seff 151807
perl: error: plugin_load_from_file: dlopen(/usr/lib64/slurm/accounting_storage_slurmdbd.so): /usr/lib64/slurm/accounting_storage_slurmdbd.so: undefined symbol: node_record_count
perl: error: Couldn't load specified plugin name for accounting_storage/slurmdbd: Dlopen of plugin file failed
perl: error: cannot create accounting_storage context for accounting_storage/slurmdbd
perl: error: plugin_load_from_file: dlopen(/usr/lib64/slurm/accounting_storage_slurmdbd.so): /usr/lib64/slurm/accounting_storage_slurmdbd.so: undefined symbol: node_record_count
perl: error: Couldn't load specified plugin name for accounting_storage/slurmdbd: Dlopen of plugin file failed
perl: error: cannot create accounting_storage context for accounting_storage/slurmdbd

Please take a closer look at the output of the diff command and the output.


With regard to my slurm upgrade. I can confirm that I followed a procedure as suggested and approved by your colleagues. For the sake of the argument I have attached a file of the upgrade plan that was followed. The cluster is stable following the upgrade and slurm database commands respond as expected.


Best regards,

David


[root@blue51 slurm]# diff /usr/bin/seff /local/software/slurm/bin/seff
13a14,15
> use Term::ANSIColor;
> #$Term::ANSIColor::AUTORESET = 1;
50c52
< $job_cond{flags} = JOBCOND_FLAG_NO_TRUNC;
---
> $job_cond{without_usage_truncation} = 1;
168,170c170,180
<     printf("CPU Efficiency: %.2f%% of %s core-walltime\n", $cpu_eff, time2str($corewalltime));
<     printf("Job Wall-clock time: %s\n", time2str($walltime));
<     if ($ntasks == 1) {
---
>      if ($cpu_eff < 50) {
>         #printf color('bold red');
>         printf("**** Please note that the CPU efficency of your job is less than 50%% ****\n");
>         printf("**** Ask the HPC team for advice -- if you need it\n");
>         printf("**** CPU Efficiency: %.2f%% of %s core-walltime\n", $cpu_eff, time2str($corewalltime));
>         printf("\n");
>      } else {
>         printf("CPU Efficiency: %.2f%% of %s core-walltime\n", $cpu_eff, time2str($corewalltime));
>      }
>      #printf("CPU Efficiency: %.2f%% of %s core-walltime\n", $cpu_eff, time2str($corewalltime));
>      if ($ntasks == 1) {
178,180c188,190
<     } else {
<         $mem_eff = 0.0;
<     }
---
> #    } else {
> #        $mem_eff = 0.0;
> #    }
188a199
>     }



________________________________
From: bugs@schedmd.com <bugs@schedmd.com>
Sent: 01 October 2018 19:09
To: Baker D.J.
Subject: [Bug 5796] Missing accounting plugins in slurm 18.08 installation


Comment # 1<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.schedmd.com%2Fshow_bug.cgi%3Fid%3D5796%23c1&data=01%7C01%7Cd.j.baker%40soton.ac.uk%7Cb1cd83024d124cee550a08d627c904d2%7C4a5378f929f44d3ebe89669d03ada9d8%7C1&sdata=87Jj1j%2FdwL6VwEzBZDAuHyMAirj5PFT2iX3bZSWhkig%3D&reserved=0> on bug 5796<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.schedmd.com%2Fshow_bug.cgi%3Fid%3D5796&data=01%7C01%7Cd.j.baker%40soton.ac.uk%7Cb1cd83024d124cee550a08d627c904d2%7C4a5378f929f44d3ebe89669d03ada9d8%7C1&sdata=WBffNT8OfhwilW6kp5DG6JI0QDX8z7ATexlI0aajchc%3D&reserved=0> from Felip Moll<mailto:felip.moll@schedmd.com>

Hi David,

Can you please describe exactly how you did perform the upgrade?

Old nodes should still be able to work exactly as new nodes since they can talk
to the controller.
What are the exact errors you are finding?

Your seff script may be the one from the old release of Slurm, so when running
from the new nodes
the calls that it does may be looking for symbols that are not in the same file
than before
(the library name has changed from libslurm to libslurmfull.so). seff uses the
Perl API.

Can you take a look at your /local/software/slurm/bin/seff and see if it is the
new or the old one?

________________________________
You are receiving this mail because:

  *   You reported the bug.

Hi David,

I was out of office until today. I read through your installation process but didn't have time to fully analyze and respond.

My though is that you still miss something in the installation regarding to seff, or there was some file not being upgraded. Maybe spec file has a mistake.

Let me check it again today/tomorrow and I will respond asap.

Hi David, hi Felip,

I can confirm this, I get the very same error, also with slurm 18.08.1.

perl: error: plugin_load_from_file: dlopen(/usr/lib64/slurm/accounting_storage_slurmdbd.so): /usr/lib64/slurm/accounting_storage_slurmdbd.so: undefined symbol: node_record_count
perl: error: Couldn't load specified plugin name for accounting_storage/slurmdbd: Dlopen of plugin file failed
perl: error: cannot create accounting_storage context for accounting_storage/slurmdbd
perl: error: plugin_load_from_file: dlopen(/usr/lib64/slurm/accounting_storage_slurmdbd.so): /usr/lib64/slurm/accounting_storage_slurmdbd.so: undefined symbol: node_record_count
perl: error: Couldn't load specified plugin name for accounting_storage/slurmdbd: Dlopen of plugin file failed
perl: error: cannot create accounting_storage context for accounting_storage/slurmdbd
Job not found.


So it looks liek a generic problem.

Don't hesitate to aks, if you need any further info from me.


with kind regards
Marcus

I can confirm the issue and working on a solution.

Hi David/Markus,

Quick update,

Another site is experiencing the same problem, see bug 5882.

We are working on a fix, but we are having issues reproducing it in certain systems, for example in my local Fedora 25 installation I can easily reproduce, but on a fresh install I can't. Can you please tell me which OS are you using?

$ lsb_release -a
$ cat /etc/*release*

Everything seems to come from commit b1ff43, in merge bbf563 in 18.08, which is a large change to analyze. For some reason a symbol which seems correctly exported is not found by the ld loader, and for now I couldn't find any duplicate export:

partial otuput from an 'LD_DEBUG=all seff <jobid>':

     14730:	symbol=node_record_count;  lookup in file=/usr/bin/perl [0]
     14730:	symbol=node_record_count;  lookup in file=/lib64/libperl.so.5.24 [0]
     14730:	symbol=node_record_count;  lookup in file=/lib64/libpthread.so.0 [0]
     14730:	symbol=node_record_count;  lookup in file=/lib64/libresolv.so.2 [0]
     14730:	symbol=node_record_count;  lookup in file=/lib64/libnsl.so.1 [0]
     14730:	symbol=node_record_count;  lookup in file=/lib64/libdl.so.2 [0]
     14730:	symbol=node_record_count;  lookup in file=/lib64/libm.so.6 [0]
     14730:	symbol=node_record_count;  lookup in file=/lib64/libcrypt.so.1 [0]
     14730:	symbol=node_record_count;  lookup in file=/lib64/libutil.so.1 [0]
     14730:	symbol=node_record_count;  lookup in file=/lib64/libc.so.6 [0]
     14730:	symbol=node_record_count;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
     14730:	symbol=node_record_count;  lookup in file=/lib64/libfreebl3.so [0]
     14730:	symbol=node_record_count;  lookup in file=/home/lipi/slurm/18.08/llagosti/lib64/perl5/auto/Slurmdb/Slurmdb.so [0]
     14730:	symbol=node_record_count;  lookup in file=/home/lipi/slurm/18.08/llagosti/lib/libslurmdb.so.33 [0]
     14730:	symbol=node_record_count;  lookup in file=/home/lipi/slurm/18.08/llagosti/lib64/perl5/auto/Slurm/Slurm.so [0]
     14730:	symbol=node_record_count;  lookup in file=/home/lipi/slurm/18.08/llagosti/lib/libslurm.so.33 [0]
     14730:	symbol=node_record_count;  lookup in file=/home/lipi/slurm/18.08/llagosti/lib/slurm/accounting_storage_slurmdbd.so [0]
     14730:	symbol=node_record_count;  lookup in file=/lib64/libpthread.so.0 [0]
     14730:	symbol=node_record_count;  lookup in file=/lib64/libc.so.6 [0]
     14730:	symbol=node_record_count;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
     14730:	/home/lipi/slurm/18.08/llagosti/lib/slurm/accounting_storage_slurmdbd.so: error: symbol lookup error: undefined symbol: node_record_count (fatal)

Will keep working on it, sorry for the delay.

Sehr geehrte Damen und Herren,



ich bin bis zum 23.10.2018 nicht erreichbar. Ihre E-Mail wird nicht weitergeleitet und kann deshalb erst danach beantwortet werden. In dringenden Fällen hilft Ihnen unser Servicedesk gerne weiter:


servicedesk@itc.rwth-aachen.de Tel.: +49 241 80 24680



mit freundlichen Grüßen
Marcus Wagner

Created attachment 8082 [details]
seff output

Hi Felip,

I have attached the seff output.

lsb_release -a:
LSB Version:    :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID: CentOS
Description:    CentOS Linux release 7.5.1804 (Core) 
Release:        7.5.1804
Codename:       Core

cat /etc/*release*:
CentOS Linux release 7.5.1804 (Core) 
Derived from Red Hat Enterprise Linux 7.5 (Source)
cat: /etc/lsb-release.d: Is a directory
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.5.1804 (Core) 
CentOS Linux release 7.5.1804 (Core) 
cpe:/o:centos:centos:7



Best
Marcus

Hi Felip,Nate,

Thank you for your work on this ticket. My apologies for my late reply. I have been

away from the university for a short break and only returned to work today. The

output you requested is appended below.

Best regards,

David

[root@red100 ~]# lsb_release -a
LSB Version:    :core-4.1-amd64:core-4.1-noarch
Distributor ID: RedHatEnterpriseServer
Description:    Red Hat Enterprise Linux Server release 7.4 (Maipo)
Release:        7.4
Codename:       Maipo
[root@red100 ~]# cat /etc/*release*
cat: /etc/lsb-release.d: Is a directory
NAME="Red Hat Enterprise Linux Server"
VERSION="7.4 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.4"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.4 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.4:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.4
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.4"
Red Hat Enterprise Linux Server release 7.4 (Maipo)
Red Hat Enterprise Linux Server release 7.4 (Maipo)
cpe:/o:redhat:enterprise_linux:7.4:ga:server





________________________________
From: bugs@schedmd.com <bugs@schedmd.com>
Sent: 25 October 2018 00:04
To: Baker D.J.
Subject: [Bug 5796] Missing accounting plugins in slurm 18.08 installation

Nate Rini<mailto:nate@schedmd.com> changed bug 5796<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.schedmd.com%2Fshow_bug.cgi%3Fid%3D5796&data=01%7C01%7Cd.j.baker%40soton.ac.uk%7C2d34c96c85f84c54a22c08d63a05048f%7C4a5378f929f44d3ebe89669d03ada9d8%7C1&sdata=37UMJktfQ4KYjgl9Hux1vPrXjkt0tAoh0Zrn0uMVUz0%3D&reserved=0>
What    Removed Added
CC              nate@schedmd.com

________________________________
You are receiving this mail because:

  *   You reported the bug.

Hi,

We've finally discovered that commit b1ff43 caused some symbols to be missing in 18.08. A patch has been created and commited with id 3d85c8f92, which is already present in 18.08.3 that came out yesterday. 

Please re-open this ticket if there are any further issues.

Thank you for reporting!

*** This ticket has been marked as a duplicate of ticket 5882 ***