Ticket 3549 - sometimes associations not propagated from slurmdbd to slurmctld
Summary: sometimes associations not propagated from slurmdbd to slurmctld
Status: RESOLVED TIMEDOUT
Alias: None
Product: Slurm
Classification: Unclassified
Component: slurmdbd (show other tickets)
Version: 16.05.9
Hardware: Linux Linux
: 4 - Minor Issue
Assignee: Tim Wickberg
QA Contact:
URL:
Depends on:
Blocks:
 
Reported: 2017-03-08 00:07 MST by Doug Jacobsen
Modified: 2019-01-02 06:34 MST (History)
0 users

See Also:
Site: NERSC
Slinky Site: ---
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
Google sites: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
NoveTech Sites: ---
Nvidia HWinf-CS Sites: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Tzag Elita Sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed:
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---


Attachments

Note You need to log in before you can comment on or make changes to this ticket.
Description Doug Jacobsen 2017-03-08 00:07:35 MST
Hello,

I have found a few cases in recent history (past few months), on edison in particular, where a user exists in the database but they are not allowed to submit jobs.

I more deeply examined a case today where a new user complained that they were unable to submit any jobs.  I found that I had to restart slurmctld to get the database and slurmctld to re-sync.

nid02305:~ # scontrol show assoc | grep sylviakl
nid02305:~ # sacctmgr show assoc where user=sylviakl -p
Cluster|Account|User|Partition|Share|GrpJobs|GrpTRES|GrpSubmit|GrpWall|GrpTRESMins|MaxJobs|MaxTRES|MaxTRESPerNode|MaxSubmit|MaxWall|MaxTRESMins|QOS|Def QOS|GrpTRESRunMins|
esedison|m1991|sylviakl||1||||||||||||debug,long,low,low_regular_0,low_regular_1,normal,normal_regular_0,normal_regular_1,preemptable,premium,premium_regular_0,premium_regular_1,scavenger,serialize|||
edison|m1991|sylviakl||1||||||||||||debug,long,low,low_regular_0,low_regular_1,normal,normal_regular_0,normal_regular_1,preemptable,premium,premium_regular_0,premium_regular_1,scavenger,serialize|||
nid02305:~ # service slurm stop
stopping slurmctld: ps                                                                                                                             done

 aux | greslurmctld is stopped
nid02305:~ # ps aux | grep slurm
root     19055  0.0  0.0   4372   584 pts/0    S+   22:46   0:00 tail -f /var/tmp/slurm/slurmctld.log
root     20362  0.0  0.0   4588   584 pts/1    S+   22:55   0:00 grep slurm
nid02305:~ # service slurm start
starting slurmctld:                                                                                                                                done

nid02305:~ # for i in $(sinfo --format=%R -h); do scontrol update part=$i state=up; done
nid02305:~ # scontrol show assoc | grep sylviakl
UserName=sylviakl(60810) DefAccount=m1991 DefWckey= AdminLevel=None
ClusterName=edison Account=m1991 UserName=sylviakl(60810) Partition= ID=3367
nid02305:~ # 


I don't see any particular entries in the slurmdbd log that pertain to this user.  The only entries in slurmctld appear to be related to the user failing to submit jobs.


The slurmdbd log doesn't have any obvious errors:
edique01:/var/tmp/slurm # tail -n 500000 slurmdbd.log | grep error
[2017-03-07T00:00:13.135] error: Connection 11 experienced an error
[2017-03-07T08:36:27.920] debug2: DBD_STEP_START: ID:4267201.3 NAME:cal_xi_mean_error.x SUBMIT:1488904418
edique01:/var/tmp/slurm # 

(we rotate slurmctld logs at midnight on edison, so the connection disruption is probably the sighup.)

What do you think might cause this?

-Doug
Comment 1 Tim Wickberg 2017-03-09 14:04:11 MST
The usual cause for this is something blocking slurmdbd -> slurmctld communication, since new association / association creation messages are pushed. It's not a common communication path, and everything else can be fully functional when this happens.

This then resolves itself when the slurmctld is restarted - as part of the startup it requests an updated copy of the association records.

Do you know when that user account was created, and have the slurmdbd logs from that point in time? Those are the ones I'd be interested in seeing.

17.02 should alleviate some of these issues - the slurmctld <-> slurmdbd connection persists now, and I believe this thus avoids issues with slurmdbd being unable to connect to slurmctld on demand. (Usually caused by some unexpected side effects from deploying firewalls between the hosts. The slurmctld -> slurmdbd breaks immediately as is an obvious problem, but people don't immediate notice the other direction is also broken and have run that way for an extended time.)
Comment 2 Doug Jacobsen 2017-03-09 14:49:48 MST
We definitely don't have a firewall between them.  I tend to have 10s to
100s of sacctmgr updates going from slurmdbd to slurmctld per day.  The
ones I notice not working are cases like this one (new user can't submit
jobs).  But most of the time everything works fine in both directions.
I'll see if I can track down a time (hopefully from the database itself...)
so I can send just a relevant segment of the 16GB unrotated slurmdbd.log.

-Doug

----
Doug Jacobsen, Ph.D.
NERSC Computer Systems Engineer
National Energy Research Scientific Computing Center <http://www.nersc.gov>
dmjacobsen@lbl.gov

------------- __o
---------- _ '\<,_
----------(_)/  (_)__________________________


On Thu, Mar 9, 2017 at 1:04 PM, <bugs@schedmd.com> wrote:

> Tim Wickberg <tim@schedmd.com> changed bug 3549
> <https://bugs.schedmd.com/show_bug.cgi?id=3549>
> What Removed Added
> Assignee support@schedmd.com tim@schedmd.com
>
> *Comment # 1 <https://bugs.schedmd.com/show_bug.cgi?id=3549#c1> on bug
> 3549 <https://bugs.schedmd.com/show_bug.cgi?id=3549> from Tim Wickberg
> <tim@schedmd.com> *
>
> The usual cause for this is something blocking slurmdbd -> slurmctld
> communication, since new association / association creation messages are
> pushed. It's not a common communication path, and everything else can be fully
> functional when this happens.
>
> This then resolves itself when the slurmctld is restarted - as part of the
> startup it requests an updated copy of the association records.
>
> Do you know when that user account was created, and have the slurmdbd logs from
> that point in time? Those are the ones I'd be interested in seeing.
>
> 17.02 should alleviate some of these issues - the slurmctld <-> slurmdbd
> connection persists now, and I believe this thus avoids issues with slurmdbd
> being unable to connect to slurmctld on demand. (Usually caused by some
> unexpected side effects from deploying firewalls between the hosts. The
> slurmctld -> slurmdbd breaks immediately as is an obvious problem, but people
> don't immediate notice the other direction is also broken and have run that way
> for an extended time.)
>
> ------------------------------
> You are receiving this mail because:
>
>    - You reported the bug.
>
>
Comment 3 Tim Wickberg 2017-03-09 16:25:46 MST
(In reply to Doug Jacobsen from comment #2)
> We definitely don't have a firewall between them.  I tend to have 10s to
> 100s of sacctmgr updates going from slurmdbd to slurmctld per day.  The
> ones I notice not working are cases like this one (new user can't submit
> jobs).  But most of the time everything works fine in both directions.
> I'll see if I can track down a time (hopefully from the database itself...)
> so I can send just a relevant segment of the 16GB unrotated slurmdbd.log.

Yeah. I assumed that wouldn't have been the issue... but one out of quite a few others is an odd issue to have. Logs from that user addition would be great if you can extract them, or if you want to crush them down with xz / bzip2 it wouldn't be the largest one I've had to hunt through...
Comment 4 Tim Wickberg 2017-04-18 21:46:37 MDT
(In reply to Tim Wickberg from comment #3)
> (In reply to Doug Jacobsen from comment #2)
> > We definitely don't have a firewall between them.  I tend to have 10s to
> > 100s of sacctmgr updates going from slurmdbd to slurmctld per day.  The
> > ones I notice not working are cases like this one (new user can't submit
> > jobs).  But most of the time everything works fine in both directions.
> > I'll see if I can track down a time (hopefully from the database itself...)
> > so I can send just a relevant segment of the 16GB unrotated slurmdbd.log.
> 
> Yeah. I assumed that wouldn't have been the issue... but one out of quite a
> few others is an odd issue to have. Logs from that user addition would be
> great if you can extract them, or if you want to crush them down with xz /
> bzip2 it wouldn't be the largest one I've had to hunt through...

Any chance to grab these? I'll follow up next week if you're out, just want to make sure I'm not the delay on this.

- Tim
Comment 5 Tim Wickberg 2017-05-03 12:58:43 MDT
(In reply to Tim Wickberg from comment #4)
> (In reply to Tim Wickberg from comment #3)
> > (In reply to Doug Jacobsen from comment #2)
> > > We definitely don't have a firewall between them.  I tend to have 10s to
> > > 100s of sacctmgr updates going from slurmdbd to slurmctld per day.  The
> > > ones I notice not working are cases like this one (new user can't submit
> > > jobs).  But most of the time everything works fine in both directions.
> > > I'll see if I can track down a time (hopefully from the database itself...)
> > > so I can send just a relevant segment of the 16GB unrotated slurmdbd.log.
> > 
> > Yeah. I assumed that wouldn't have been the issue... but one out of quite a
> > few others is an odd issue to have. Logs from that user addition would be
> > great if you can extract them, or if you want to crush them down with xz /
> > bzip2 it wouldn't be the largest one I've had to hunt through...
> 
> Any chance to grab these? I'll follow up next week if you're out, just want
> to make sure I'm not the delay on this.
> 
> - Tim

Ping.
Comment 6 Tim Wickberg 2017-05-23 18:41:44 MDT
> Ping.

Not sure if this one is still an active concern, but you'd promised some logs a couple of months ago... marking resolved/timedout for now, please reopen if you'd like to continue to pursue this one.

- Tim
Comment 7 Doug Jacobsen 2017-05-23 23:13:57 MDT
Yeah, too bad, the logs really weren't informative and I could never get
the gigabytes of slurmdbd logs to show anything useful.  Then that server
had an issue and the logs were lost.  Sorry about that.   Anyway, I've had
a few more instances where I've had to restart slurmctld to get the
slurmctld cache to update properly, but haven't found any reasonable
reporter yet. (It's a rare issue, and usually occurs when I'm overloaded
with other things)

----
Doug Jacobsen, Ph.D.
NERSC Computer Systems Engineer
National Energy Research Scientific Computing Center <http://www.nersc.gov>
dmjacobsen@lbl.gov

------------- __o
---------- _ '\<,_
----------(_)/  (_)__________________________


On Tue, May 23, 2017 at 5:41 PM, <bugs@schedmd.com> wrote:

> Tim Wickberg <tim@schedmd.com> changed bug 3549
> <https://bugs.schedmd.com/show_bug.cgi?id=3549>
> What Removed Added
> Status UNCONFIRMED RESOLVED
> Resolution --- TIMEDOUT
>
> *Comment # 6 <https://bugs.schedmd.com/show_bug.cgi?id=3549#c6> on bug
> 3549 <https://bugs.schedmd.com/show_bug.cgi?id=3549> from Tim Wickberg
> <tim@schedmd.com> *
>
> > Ping.
>
> Not sure if this one is still an active concern, but you'd promised some logs a
> couple of months ago... marking resolved/timedout for now, please reopen if
> you'd like to continue to pursue this one.
>
> - Tim
>
> ------------------------------
> You are receiving this mail because:
>
>    - You reported the bug.
>
>
Comment 8 Tim Wickberg 2017-05-23 23:27:11 MDT
Okay - leaving at 'timedout' for now; you can always reopen if you catch it happening again.

Thanks for you patience - I know troubleshooting these takes a back seat to just getting things operational at times, but I also want to make sure I'm not bugging you constantly on relatively unimportant issues.

cheers,
- Tim