Ticket 9827

I misspoke earlier.  This is v20.02.4, not 20.02.5.  I've updated the bug to reflect this.

Could you please load the core to gdb and execute:
>(gdb) p $_siginfo
>(gdb) ptype $_siginfo

Is it possible that there was a change in slurm binaries stored on NFS or change in files like passwd/shadow refered by nss_compat?  

cheers,
Marcin

Created attachment 15911 [details]
output of requested "p $_signinfo" and "ptype $_signinfo"

I've attached the requested output here.  Let me know if you need additional information.

As far as I know, there should not have been any changes to the slurm binaries without a restart.  As you know from Bug 9726, we did upgrade to 20.02.4, but that was days before the crash, and we restarted both slurmctld and slurmdbd after the upgrade.  Also, the storage location is local to the node, not NFS.

Could you please execute build and execute the following test program on the host:
>  1 #include<stdio.h>                                                                
>  2 #include <unistd.h>                                                              
>  3                                                                                  
>  4 void main(void) {                                                                
>  5         printf("Buffer size is:%d",_SC_GETGR_R_SIZE_MAX);                        
>  6 }  
It should compile with just `gcc /tmp/testProgram.c -o /tmp/output`.


The gdb output shows that sigbus was received with si_code=2 which means BUS_ADRERR - Access to a non-existent area of a memory object[1].

This error is most likely(in general) comming from a failure in accessing mmaped file - that's why I asked about NFS which is probably not the case based on your reply. Looking at the backtrace the active thread was outside of Slurm code executing something like:
#getent passwd vmosquer
Is the user active? Do you store any user database file on NFS? 

Probably not the case, but... There was a bug a few years ago (CentOS6) where this si_code was set for all cases when hudge pages were not disabled, what's your os/kernel version? The output suggests that her shell is set to /bin/false. What is your nsswitch user information source for the user?

cheers,
Marcin

Marcin,

I apologize for the delay.  We got distracted by another systems issue.

root@sched1:/tmp# ./testProgram 
Buffer size is:69
root@sched1:/tmp#

As far as the rest goes, the slurm binary is definitely local to the host.  The only NFS involved in normal operation, is in the "StateSaveLocation" path.

Having said that, the user credentials (like vmosquer, who is an active user), are populated into the local /etc/{passwd,shadow,group} files via a script.  It's theoretically possible that it happened to be clobbering one of those files, at the same time as the sigbus, I suppose.  Though it's not clear to me if that scenario would even lead to a sigbus in the first place.

 Lloyd,

Looking further into glibc/nss_compat code I think that you hit the bug there. It looks like getpwnam_r (which should be thread safe) makes use of fgets_unlocked(which is not thread safe). I'll report this in glibc bugzilla and add a link to it here.

Will that be fine if I'll attach the backtrace you shared with us there?

cheers
Marcin

 Lloyd,

I've anonymized all real data in the thread 1 backtrace and attached it to the case in glibc buzilla:
https://sourceware.org/bugzilla/show_bug.cgi?id=26713

I hope you don't have anything against that. The attached information doesn't have any personal info nor hostname/ip addesses. It's just a sequence of functions called in glibc.

cheers,
Marcin

Marcin,

I don't see any problem with sharing the information.  Sorry that I missed your previous message.  Too many other issues to deal with, I guess.

Lloyd

 Lloyd,


Do you agree that we can read Florian Weimer(RedHat) comment[1] as a confirmation that the bug is actually outside of Slurm and we can close this bug report?

cheers,
Marcin

[1]https://sourceware.org/bugzilla/show_bug.cgi?id=26713#c1

Marcin,

I'm not sure I'm the best qualified to discuss the underlying bug.  But if you believe that the problem exists outside of Slurm, and you've opened the appropriate upstream bug, then I'm certainly not going to contradict you.