Hello, I have found a few cases in recent history (past few months), on edison in particular, where a user exists in the database but they are not allowed to submit jobs. I more deeply examined a case today where a new user complained that they were unable to submit any jobs. I found that I had to restart slurmctld to get the database and slurmctld to re-sync. nid02305:~ # scontrol show assoc | grep sylviakl nid02305:~ # sacctmgr show assoc where user=sylviakl -p Cluster|Account|User|Partition|Share|GrpJobs|GrpTRES|GrpSubmit|GrpWall|GrpTRESMins|MaxJobs|MaxTRES|MaxTRESPerNode|MaxSubmit|MaxWall|MaxTRESMins|QOS|Def QOS|GrpTRESRunMins| esedison|m1991|sylviakl||1||||||||||||debug,long,low,low_regular_0,low_regular_1,normal,normal_regular_0,normal_regular_1,preemptable,premium,premium_regular_0,premium_regular_1,scavenger,serialize||| edison|m1991|sylviakl||1||||||||||||debug,long,low,low_regular_0,low_regular_1,normal,normal_regular_0,normal_regular_1,preemptable,premium,premium_regular_0,premium_regular_1,scavenger,serialize||| nid02305:~ # service slurm stop stopping slurmctld: ps done aux | greslurmctld is stopped nid02305:~ # ps aux | grep slurm root 19055 0.0 0.0 4372 584 pts/0 S+ 22:46 0:00 tail -f /var/tmp/slurm/slurmctld.log root 20362 0.0 0.0 4588 584 pts/1 S+ 22:55 0:00 grep slurm nid02305:~ # service slurm start starting slurmctld: done nid02305:~ # for i in $(sinfo --format=%R -h); do scontrol update part=$i state=up; done nid02305:~ # scontrol show assoc | grep sylviakl UserName=sylviakl(60810) DefAccount=m1991 DefWckey= AdminLevel=None ClusterName=edison Account=m1991 UserName=sylviakl(60810) Partition= ID=3367 nid02305:~ # I don't see any particular entries in the slurmdbd log that pertain to this user. The only entries in slurmctld appear to be related to the user failing to submit jobs. The slurmdbd log doesn't have any obvious errors: edique01:/var/tmp/slurm # tail -n 500000 slurmdbd.log | grep error [2017-03-07T00:00:13.135] error: Connection 11 experienced an error [2017-03-07T08:36:27.920] debug2: DBD_STEP_START: ID:4267201.3 NAME:cal_xi_mean_error.x SUBMIT:1488904418 edique01:/var/tmp/slurm # (we rotate slurmctld logs at midnight on edison, so the connection disruption is probably the sighup.) What do you think might cause this? -Doug
The usual cause for this is something blocking slurmdbd -> slurmctld communication, since new association / association creation messages are pushed. It's not a common communication path, and everything else can be fully functional when this happens. This then resolves itself when the slurmctld is restarted - as part of the startup it requests an updated copy of the association records. Do you know when that user account was created, and have the slurmdbd logs from that point in time? Those are the ones I'd be interested in seeing. 17.02 should alleviate some of these issues - the slurmctld <-> slurmdbd connection persists now, and I believe this thus avoids issues with slurmdbd being unable to connect to slurmctld on demand. (Usually caused by some unexpected side effects from deploying firewalls between the hosts. The slurmctld -> slurmdbd breaks immediately as is an obvious problem, but people don't immediate notice the other direction is also broken and have run that way for an extended time.)
We definitely don't have a firewall between them. I tend to have 10s to 100s of sacctmgr updates going from slurmdbd to slurmctld per day. The ones I notice not working are cases like this one (new user can't submit jobs). But most of the time everything works fine in both directions. I'll see if I can track down a time (hopefully from the database itself...) so I can send just a relevant segment of the 16GB unrotated slurmdbd.log. -Doug ---- Doug Jacobsen, Ph.D. NERSC Computer Systems Engineer National Energy Research Scientific Computing Center <http://www.nersc.gov> dmjacobsen@lbl.gov ------------- __o ---------- _ '\<,_ ----------(_)/ (_)__________________________ On Thu, Mar 9, 2017 at 1:04 PM, <bugs@schedmd.com> wrote: > Tim Wickberg <tim@schedmd.com> changed bug 3549 > <https://bugs.schedmd.com/show_bug.cgi?id=3549> > What Removed Added > Assignee support@schedmd.com tim@schedmd.com > > *Comment # 1 <https://bugs.schedmd.com/show_bug.cgi?id=3549#c1> on bug > 3549 <https://bugs.schedmd.com/show_bug.cgi?id=3549> from Tim Wickberg > <tim@schedmd.com> * > > The usual cause for this is something blocking slurmdbd -> slurmctld > communication, since new association / association creation messages are > pushed. It's not a common communication path, and everything else can be fully > functional when this happens. > > This then resolves itself when the slurmctld is restarted - as part of the > startup it requests an updated copy of the association records. > > Do you know when that user account was created, and have the slurmdbd logs from > that point in time? Those are the ones I'd be interested in seeing. > > 17.02 should alleviate some of these issues - the slurmctld <-> slurmdbd > connection persists now, and I believe this thus avoids issues with slurmdbd > being unable to connect to slurmctld on demand. (Usually caused by some > unexpected side effects from deploying firewalls between the hosts. The > slurmctld -> slurmdbd breaks immediately as is an obvious problem, but people > don't immediate notice the other direction is also broken and have run that way > for an extended time.) > > ------------------------------ > You are receiving this mail because: > > - You reported the bug. > >
(In reply to Doug Jacobsen from comment #2) > We definitely don't have a firewall between them. I tend to have 10s to > 100s of sacctmgr updates going from slurmdbd to slurmctld per day. The > ones I notice not working are cases like this one (new user can't submit > jobs). But most of the time everything works fine in both directions. > I'll see if I can track down a time (hopefully from the database itself...) > so I can send just a relevant segment of the 16GB unrotated slurmdbd.log. Yeah. I assumed that wouldn't have been the issue... but one out of quite a few others is an odd issue to have. Logs from that user addition would be great if you can extract them, or if you want to crush them down with xz / bzip2 it wouldn't be the largest one I've had to hunt through...
(In reply to Tim Wickberg from comment #3) > (In reply to Doug Jacobsen from comment #2) > > We definitely don't have a firewall between them. I tend to have 10s to > > 100s of sacctmgr updates going from slurmdbd to slurmctld per day. The > > ones I notice not working are cases like this one (new user can't submit > > jobs). But most of the time everything works fine in both directions. > > I'll see if I can track down a time (hopefully from the database itself...) > > so I can send just a relevant segment of the 16GB unrotated slurmdbd.log. > > Yeah. I assumed that wouldn't have been the issue... but one out of quite a > few others is an odd issue to have. Logs from that user addition would be > great if you can extract them, or if you want to crush them down with xz / > bzip2 it wouldn't be the largest one I've had to hunt through... Any chance to grab these? I'll follow up next week if you're out, just want to make sure I'm not the delay on this. - Tim
(In reply to Tim Wickberg from comment #4) > (In reply to Tim Wickberg from comment #3) > > (In reply to Doug Jacobsen from comment #2) > > > We definitely don't have a firewall between them. I tend to have 10s to > > > 100s of sacctmgr updates going from slurmdbd to slurmctld per day. The > > > ones I notice not working are cases like this one (new user can't submit > > > jobs). But most of the time everything works fine in both directions. > > > I'll see if I can track down a time (hopefully from the database itself...) > > > so I can send just a relevant segment of the 16GB unrotated slurmdbd.log. > > > > Yeah. I assumed that wouldn't have been the issue... but one out of quite a > > few others is an odd issue to have. Logs from that user addition would be > > great if you can extract them, or if you want to crush them down with xz / > > bzip2 it wouldn't be the largest one I've had to hunt through... > > Any chance to grab these? I'll follow up next week if you're out, just want > to make sure I'm not the delay on this. > > - Tim Ping.
> Ping. Not sure if this one is still an active concern, but you'd promised some logs a couple of months ago... marking resolved/timedout for now, please reopen if you'd like to continue to pursue this one. - Tim
Yeah, too bad, the logs really weren't informative and I could never get the gigabytes of slurmdbd logs to show anything useful. Then that server had an issue and the logs were lost. Sorry about that. Anyway, I've had a few more instances where I've had to restart slurmctld to get the slurmctld cache to update properly, but haven't found any reasonable reporter yet. (It's a rare issue, and usually occurs when I'm overloaded with other things) ---- Doug Jacobsen, Ph.D. NERSC Computer Systems Engineer National Energy Research Scientific Computing Center <http://www.nersc.gov> dmjacobsen@lbl.gov ------------- __o ---------- _ '\<,_ ----------(_)/ (_)__________________________ On Tue, May 23, 2017 at 5:41 PM, <bugs@schedmd.com> wrote: > Tim Wickberg <tim@schedmd.com> changed bug 3549 > <https://bugs.schedmd.com/show_bug.cgi?id=3549> > What Removed Added > Status UNCONFIRMED RESOLVED > Resolution --- TIMEDOUT > > *Comment # 6 <https://bugs.schedmd.com/show_bug.cgi?id=3549#c6> on bug > 3549 <https://bugs.schedmd.com/show_bug.cgi?id=3549> from Tim Wickberg > <tim@schedmd.com> * > > > Ping. > > Not sure if this one is still an active concern, but you'd promised some logs a > couple of months ago... marking resolved/timedout for now, please reopen if > you'd like to continue to pursue this one. > > - Tim > > ------------------------------ > You are receiving this mail because: > > - You reported the bug. > >
Okay - leaving at 'timedout' for now; you can always reopen if you catch it happening again. Thanks for you patience - I know troubleshooting these takes a back seat to just getting things operational at times, but I also want to make sure I'm not bugging you constantly on relatively unimportant issues. cheers, - Tim