Ticket 2395

Summary:	sacct fails to display information
Product:	Slurm	Reporter:	Charles Johnson <charles.johnson>
Component:	User Commands	Assignee:	Tim Wickberg <tim>
Status:	RESOLVED INFOGIVEN	QA Contact:
Severity:	3 - Medium Impact
Priority:	---	CC:	davide.vanzo, sean, will
Version:	15.08.7
Hardware:	Linux
OS:	Linux
Site:	Vanderbilt	Alineos Sites:	---
Atos/Eviden Sites:	---	Confidential Site:	---
Coreweave sites:	---	Cray Sites:	---
DS9 clusters:	---	HPCnow Sites:	---
HPE Sites:	---	IBM Sites:	---
NOAA SIte:	---	OCF Sites:	---
Recursion Pharma Sites:	---	SFW Sites:	---
SNIC sites:	---	Linux Distro:	---
Machine Name:		CLE Version:
Version Fixed:		Target Release:	---
DevPrio:	---	Emory-Cloud Sites:	---
Attachments:	slurm.conf slurmctld.log slurmdb.conf slurmdbd.log SlurmUpdate.pl (perl script) GridSlurmUpdate.pl (perl script)

Description Charles Johnson 2016-01-28 04:38:48 MST

Submitted a job which is currently running, and has been for 90 minutes, or so:

# squeue -u johns276
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
           6953472 productio slurm1.t johns276  R    1:26:06      1 vmp567

sacct shows no information about the job:

 sacct --jobs 6953472
       JobID    JobName  Partition    Account  AllocCPUS      State ExitCode 
------------ ---------- ---------- ---------- ---------- ---------- -------- 

We have a number of in-house tools for users that parse the output from sacct. Those tools now fail. However, they do not fail all of the time.

sacct --jobs 6906149
       JobID    JobName  Partition    Account  AllocCPUS      State ExitCode 
------------ ---------- ---------- ---------- ---------- ---------- -------- 
6906149      tree_2.sl+ production   rokaslab         10    RUNNING      0:0 

But for other job, sacct fails.

sacct --jobs 6951273
       JobID    JobName  Partition    Account  AllocCPUS      State ExitCode 
------------ ---------- ---------- ---------- ---------- ---------- -------- 

Any help or insights would be welcome.

Charles

Comment 1 Tim Wickberg 2016-01-28 05:37:11 MST

Can you attach slurmdbd.log and slurmctld.log for the relevant period? I'm curious if there are any messages in there about communication issues.

Running jobs should be marked as such in the database, although there can be some minor delays updating those records depending on your configuration. A copy of your slurm.conf would also be helpful.

Are any jobs newer than 6953472 recorded in the database, or is there some point in time that info stopped flowing correctly? The slurmctld process will spool a certain number of records in case of problems, but that should be getting flushed out if there was some temporary problem.

- Tim

Comment 2 Charles Johnson 2016-01-28 06:01:25 MST

Created attachment 2661 [details]
slurm.conf

Comment 3 Charles Johnson 2016-01-28 06:02:28 MST

Created attachment 2662 [details]
slurmctld.log

Comment 4 Charles Johnson 2016-01-28 06:03:08 MST

Created attachment 2663 [details]
slurmdb.conf

Comment 5 Charles Johnson 2016-01-28 06:03:56 MST

Created attachment 2664 [details]
slurmdbd.log

Comment 6 Charles Johnson 2016-01-28 06:09:20 MST

Jobs newer show up:

sacct --jobs 6954348
       JobID    JobName  Partition    Account  AllocCPUS      State ExitCode 
------------ ---------- ---------- ---------- ---------- ---------- -------- 
6954348      PROD16687+ production  mass_spec          8    RUNNING      0:0 

Now my job shows up:

sacct --jobs 6953472
       JobID    JobName  Partition    Account  AllocCPUS      State ExitCode 
------------ ---------- ---------- ---------- ---------- ---------- -------- 
6953472      slurm1.te+ production      accre          1    RUNNING      0:0

Comment 7 Tim Wickberg 2016-01-28 07:08:27 MST

Are you using the JobComp logs? I'd suggest turning that off - we recommend sticking with slurmdbd's accounting tools unless you have some specific reason for using both. Otherwise you just generate more traffic to your MySQL server.

You've got a number of errors like:
error: find_node_record: lookup failure for vmp698

Are you running a slurmd on the node while not having an entry in slurm.conf? I'm guessing the numbers were sequential originally but you've dropped them out of the config for hardware or other reasons?

Can you increase the debug level on slurmdbd to Debug2 and run it for a while? I'm not seeing any obvious issues, although I find it odd that root is reconnecting twice every five minutes. Do you have some cron job launching sacct/sacctmgr every five minutes that could be causing some interference?

Comment 8 Charles Johnson 2016-02-01 05:19:49 MST

It has begun happening again. sacct shows no output but scontrol does:

[root@vmps10 ~]# sacct -j 6990489
       JobID    JobName  Partition    Account  AllocCPUS      State ExitCode 
------------ ---------- ---------- ---------- ---------- ---------- -------- 


[root@vmps10 ~]# scontrol show jobs 6990489
JobId=6990489 JobName=CALSER-x-2101206-x-2101206_sess1-x-11-x-ON_MR_segmentation_v2.slurm
   UserId=vuiiscci(59223) GroupId=h_vuiis(36052)
   Priority=91 Nice=0 Account=h_vuiis QOS=normal
   JobState=RUNNING Reason=None Dependency=(null)
   Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
   RunTime=00:50:54 TimeLimit=2-00:00:00 TimeMin=N/A
   SubmitTime=2016-02-01T11:09:28 EligibleTime=2016-02-01T11:09:28
   StartTime=2016-02-01T12:25:51 EndTime=2016-02-03T12:25:51
   PreemptTime=None SuspendTime=None SecsPreSuspend=0
   Partition=production AllocNode:Sid=elm:7857
   ReqNodeList=(null) ExcNodeList=(null)
   NodeList=vmp380
   BatchHost=vmp380
   NumNodes=1 NumCPUs=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
   TRES=cpu=1,mem=8192,node=1
   Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
   MinCPUsNode=1 MinMemoryNode=8G MinTmpDiskNode=0
   Features=(null) Gres=(null) Reservation=(null)
   Shared=OK Contiguous=0 Licenses=(null) Network=(null)
   Command=/scratch/vuiiscci/Spider_Upload_Dir/PBS/CALSER-x-2101206-x-2101206_sess1-x-11-x-ON_MR_segmentation_v2.slurm
   WorkDir=/gpfs22/home/vuiiscci
   StdErr=/scratch/vuiiscci/Spider_Upload_Dir/OUTLOG/CALSER-x-2101206-x-2101206_sess1-x-11-x-ON_MR_segmentation_v2.output
   StdIn=/dev/null
   StdOut=/scratch/vuiiscci/Spider_Upload_Dir/OUTLOG/CALSER-x-2101206-x-2101206_sess1-x-11-x-ON_MR_segmentation_v2.output
   Power= SICP=0

Comment 9 Charles Johnson 2016-02-01 05:24:17 MST

We are aloso getting the follwing messages in the slurmctld log file:

eb  1 12:20:25 slurmsched1 slurmctld[32377]: error: Node ping apparently hung, many nodes may be DOWN or configured SlurmdTimeout should be increased
Feb  1 12:22:43 slurmsched1 slurmctld[32377]: error: Nodes vmp[302-305,364-366,368,370,389,1202] not responding
Feb  1 12:23:15 slurmsched1 slurmctld[32377]: error: slurm_receive_msgs: Socket timed out on send/recv operation
Feb  1 12:24:45 slurmsched1 slurmctld[32377]: error: slurm_receive_msgs: Socket timed out on send/recv operation
Feb  1 12:24:45 slurmsched1 slurmctld[32377]: error: slurm_receive_msgs: Socket timed out on send/recv operation
Feb  1 12:24:45 slurmsched1 slurmctld[32377]: error: slurm_receive_msgs: Socket timed out on send/recv operation
Feb  1 12:24:45 slurmsched1 slurmctld[32377]: error: slurm_receive_msgs: Socket timed out on send/recv operation
Feb  1 12:24:45 slurmsched1 slurmctld[32377]: error: slurm_receive_msgs: Socket timed out on send/recv operation
Feb  1 12:24:45 slurmsched1 slurmctld[32377]: error: slurm_receive_msgs: Socket timed out on send/recv operation

Comment 10 Tim Wickberg 2016-02-01 08:12:35 MST

Hey Charles -

I never saw a response to a few of these questions, I'm still curious if they may shed some light on the issue:

(In reply to Tim Wickberg from comment #7)
> Are you using the JobComp logs? I'd suggest turning that off - we recommend
> sticking with slurmdbd's accounting tools unless you have some specific
> reason for using both. Otherwise you just generate more traffic to your
> MySQL server.
> 
> You've got a number of errors like:
> error: find_node_record: lookup failure for vmp698
> 
> Are you running a slurmd on the node while not having an entry in
> slurm.conf? I'm guessing the numbers were sequential originally but you've
> dropped them out of the config for hardware or other reasons?
> 
> Can you increase the debug level on slurmdbd to Debug2 and run it for a
> while? I'm not seeing any obvious issues, although I find it odd that root
> is reconnecting twice every five minutes. Do you have some cron job
> launching sacct/sacctmgr every five minutes that could be causing some
> interference?


Adding a few more:

- Are there any communication issues within the cluster? Are those affected nodes (vmp[302-305,364-366,368,370,389,1202]) responding properly, or were they down for some reason? (Crash, OOM, overloaded?)

- One thing you can do to lighten the load on Slurmdbd is to set CommitDelay=1 in slurmdbd.conf. This means slurmdbd will cache up to a second of records, and then load them in to MySQL in a single transaction which can significantly improve performance.

Comment 11 Will French 2016-02-01 09:13:26 MST

Hi Tim,

Charles has left for the day so I'll try to tackle these.

(In reply to Tim Wickberg from comment #10)
> Hey Charles -
> 
> I never saw a response to a few of these questions, I'm still curious if
> they may shed some light on the issue:
> 
> (In reply to Tim Wickberg from comment #7)
> > Are you using the JobComp logs? I'd suggest turning that off - we recommend
> > sticking with slurmdbd's accounting tools unless you have some specific
> > reason for using both. Otherwise you just generate more traffic to your
> > MySQL server.

We finally turned these off last week.


> > You've got a number of errors like:
> > error: find_node_record: lookup failure for vmp698
> > 
> > Are you running a slurmd on the node while not having an entry in
> > slurm.conf? I'm guessing the numbers were sequential originally but you've
> > dropped them out of the config for hardware or other reasons?

This did turn out to be an issue. There were several nodes that were running slurmd but were not listed in slurm.conf. This has been corrected in the last week.

> > Can you increase the debug level on slurmdbd to Debug2 and run it for a
> > while? I'm not seeing any obvious issues, although I find it odd that root
> > is reconnecting twice every five minutes. Do you have some cron job
> > launching sacct/sacctmgr every five minutes that could be causing some
> > interference?

I know Charles was looking into this but I don't recall whether he ever identified the source. I'll let him comment tomorrow.

 
> Adding a few more:
> 
> - Are there any communication issues within the cluster? Are those affected
> nodes (vmp[302-305,364-366,368,370,389,1202]) responding properly, or were
> they down for some reason? (Crash, OOM, overloaded?)


slurmdbd crashed Friday morning with: 

[2016-01-29T09:54:18.999] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:197024 IP:10.0.20.242 CONN:7
[2016-01-29T09:54:19.772] Terminate signal (SIGINT or SIGTERM) received
[2016-01-29T09:54:20.304] Unable to remove pidfile '/var/tmp/slurm/slurmdbd.pid': No such file or directory
[2016-01-29T09:54:30.378] error: Unable to open pidfile `/var/tmp/slurm/slurmdbd.pid': No such file or directory

Somehow we didn't catch this until this (Monday) morning, at which point we restarted slurmdbd (I believe it took multiple tries). We've looked through the logs and cannot find any clues about why slurmdbd was killed. 

Next, in the early afternoon, we noticed that several (~30?) nodes were in a down* state. Interestingly, slurmd was running on these nodes and the nodes were accessible via ssh and responded via ping. Load was not high on the nodes. We looked through the logs on a handful of the nodes and saw nothing that appeared problematic. We tried a "service slurmd restart" which did not restart the slurmd process; it appeared to be hung. We ended up needing to do a kill -9 on the original slurmd process at which point the nodes immediately began talking to slurmctld and became available for scheduling. Please recall that we did upgrade from 14.11.10 to 15.08.7 on Wednesday. Any ideas? 

> - One thing you can do to lighten the load on Slurmdbd is to set
> CommitDelay=1 in slurmdbd.conf. This means slurmdbd will cache up to a
> second of records, and then load them in to MySQL in a single transaction
> which can significantly improve performance.

We may try this in the morning.

Comment 12 Tim Wickberg 2016-02-02 07:44:18 MST

> > > Can you increase the debug level on slurmdbd to Debug2 and run it for a
> > > while? I'm not seeing any obvious issues, although I find it odd that root
> > > is reconnecting twice every five minutes. Do you have some cron job
> > > launching sacct/sacctmgr every five minutes that could be causing some
> > > interference?
> 
> I know Charles was looking into this but I don't recall whether he ever
> identified the source. I'll let him comment tomorrow.

I'd still like to see that when you get a chance.

> > Adding a few more:
> > 
> > - Are there any communication issues within the cluster? Are those affected
> > nodes (vmp[302-305,364-366,368,370,389,1202]) responding properly, or were
> > they down for some reason? (Crash, OOM, overloaded?)
> 
> 
> slurmdbd crashed Friday morning with: 
> 
> [2016-01-29T09:54:18.999] debug:  DBD_INIT: CLUSTER:accre VERSION:7424
> UID:197024 IP:10.0.20.242 CONN:7
> [2016-01-29T09:54:19.772] Terminate signal (SIGINT or SIGTERM) received
> [2016-01-29T09:54:20.304] Unable to remove pidfile
> '/var/tmp/slurm/slurmdbd.pid': No such file or directory
> [2016-01-29T09:54:30.378] error: Unable to open pidfile
> `/var/tmp/slurm/slurmdbd.pid': No such file or directory
> 
> Somehow we didn't catch this until this (Monday) morning, at which point we
> restarted slurmdbd (I believe it took multiple tries). We've looked through
> the logs and cannot find any clues about why slurmdbd was killed. 

... that's unusual. Something is definitely terminating us, and the missing pid file is also concerning.

Do you have logrotate or something else running on the slurmdbd host that may have caused this?

What OS is slurmdbd running on? I do recall seeing some issues with systemd, and we did change the example service scripts back in November.

> Next, in the early afternoon, we noticed that several (~30?) nodes were in a
> down* state. Interestingly, slurmd was running on these nodes and the nodes
> were accessible via ssh and responded via ping. Load was not high on the
> nodes. We looked through the logs on a handful of the nodes and saw nothing
> that appeared problematic. We tried a "service slurmd restart" which did not
> restart the slurmd process; it appeared to be hung. We ended up needing to
> do a kill -9 on the original slurmd process at which point the nodes
> immediately began talking to slurmctld and became available for scheduling.
> Please recall that we did upgrade from 14.11.10 to 15.08.7 on Wednesday. Any
> ideas? 

Did anything else change with that update? Did you switch OS versions at the same time, or change how Slurm is being launched?

> > - One thing you can do to lighten the load on Slurmdbd is to set
> > CommitDelay=1 in slurmdbd.conf. This means slurmdbd will cache up to a
> > second of records, and then load them in to MySQL in a single transaction
> > which can significantly improve performance.
> 
> We may try this in the morning.

Okay - this is still one of the first quick-fixes we usually recommend. I'd be curious if that changes anything for you, or if something else is still interacting unkindly with Slurm.

Comment 13 Charles Johnson 2016-02-02 08:01:36 MST

Our log files are pretty small. Here is today's

[2016-02-02T08:30:12.824] debug:  auth plugin for Munge (http://code.google.com/p/munge/) loaded
[2016-02-02T08:30:13.339] Accounting storage MYSQL plugin loaded
[2016-02-02T08:30:13.905] debug:  post user: couldn't get a uid for user abbottpt
[2016-02-02T08:30:13.907] debug:  post user: couldn't get a uid for user abdulsmh
[2016-02-02T08:30:13.909] debug:  post user: couldn't get a uid for user abraha1
[2016-02-02T08:30:13.929] debug:  post user: couldn't get a uid for user arnoldst
[2016-02-02T08:30:13.931] debug:  post user: couldn't get a uid for user arrivirs
[2016-02-02T08:30:13.936] debug:  post user: couldn't get a uid for user artolal
[2016-02-02T08:30:13.940] debug:  post user: couldn't get a uid for user asmanaj
[2016-02-02T08:30:13.944] debug:  post user: couldn't get a uid for user aughtmsl
[2016-02-02T08:30:13.953] debug:  post user: couldn't get a uid for user baiy
[2016-02-02T08:30:13.966] debug:  post user: couldn't get a uid for user baumsh
[2016-02-02T08:30:13.968] debug:  post user: couldn't get a uid for user beckerm
[2016-02-02T08:30:13.996] debug:  post user: couldn't get a uid for user bochenj
[2016-02-02T08:30:14.001] debug:  post user: couldn't get a uid for user bradymv
[2016-02-02T08:30:14.018] debug:  post user: couldn't get a uid for user brubrej
[2016-02-02T08:30:14.021] debug:  post user: couldn't get a uid for user brumgacl
[2016-02-02T08:30:14.027] debug:  post user: couldn't get a uid for user bulenb
[2016-02-02T08:30:14.038] debug:  post user: couldn't get a uid for user cainlt
[2016-02-02T08:30:14.040] debug:  post user: couldn't get a uid for user cairesg
[2016-02-02T08:30:14.082] debug:  post user: couldn't get a uid for user chattei1
[2016-02-02T08:30:14.089] debug:  post user: couldn't get a uid for user chengf
[2016-02-02T08:30:14.108] debug:  post user: couldn't get a uid for user chhokrad
[2016-02-02T08:30:14.128] debug:  post user: couldn't get a uid for user cohendn
[2016-02-02T08:30:14.142] debug:  post user: couldn't get a uid for user coronadw
[2016-02-02T08:30:14.146] debug:  post user: couldn't get a uid for user cowanjd
[2016-02-02T08:30:14.155] debug:  post user: couldn't get a uid for user cyrk
[2016-02-02T08:30:14.157] debug:  post user: couldn't get a uid for user dalebl
[2016-02-02T08:30:14.163] debug:  post user: couldn't get a uid for user datterrd
[2016-02-02T08:30:14.181] debug:  post user: couldn't get a uid for user delahar
[2016-02-02T08:30:14.187] debug:  post user: couldn't get a uid for user demapabt
[2016-02-02T08:30:14.191] debug:  post user: couldn't get a uid for user denieama
[2016-02-02T08:30:14.194] debug:  post user: couldn't get a uid for user detheram
[2016-02-02T08:30:14.198] debug:  post user: couldn't get a uid for user dhingn1
[2016-02-02T08:30:14.202] debug:  post user: couldn't get a uid for user dossanfm
[2016-02-02T08:30:14.227] debug:  post user: couldn't get a uid for user elmoremh
[2016-02-02T08:30:14.231] debug:  post user: couldn't get a uid for user esparzma
[2016-02-02T08:30:14.239] debug:  post user: couldn't get a uid for user fanq
[2016-02-02T08:30:14.241] debug:  post user: couldn't get a uid for user farooa1
[2016-02-02T08:30:14.243] debug:  post user: couldn't get a uid for user federsj
[2016-02-02T08:30:14.245] debug:  post user: couldn't get a uid for user fedoseai
[2016-02-02T08:30:14.257] debug:  post user: couldn't get a uid for user flanagwh
[2016-02-02T08:30:14.304] debug:  post user: couldn't get a uid for user floresbr
[2016-02-02T08:30:14.306] debug:  post user: couldn't get a uid for user folstejr
[2016-02-02T08:30:14.312] debug:  post user: couldn't get a uid for user fosterd4
[2016-02-02T08:30:14.324] debug:  post user: couldn't get a uid for user fuseinh
[2016-02-02T08:30:14.330] debug:  post user: couldn't get a uid for user gail
[2016-02-02T08:30:14.361] debug:  post user: couldn't get a uid for user gillnk
[2016-02-02T08:30:14.363] debug:  post user: couldn't get a uid for user giltnajm
[2016-02-02T08:30:14.369] debug:  post user: couldn't get a uid for user gitschbl
[2016-02-02T08:30:14.372] debug:  post user: couldn't get a uid for user glassaw
[2016-02-02T08:30:14.374] debug:  post user: couldn't get a uid for user gletterb
[2016-02-02T08:30:14.407] debug:  post user: couldn't get a uid for user haleyjd
[2016-02-02T08:30:14.424] debug:  post user: couldn't get a uid for user hawkjl
[2016-02-02T08:30:14.431] debug:  post user: couldn't get a uid for user hebertm1
[2016-02-02T08:30:14.441] debug:  post user: couldn't get a uid for user herbisdr
[2016-02-02T08:30:14.451] debug:  post user: couldn't get a uid for user hlushasp
[2016-02-02T08:30:14.462] debug:  post user: couldn't get a uid for user holtj2
[2016-02-02T08:30:14.487] debug:  post user: couldn't get a uid for user hwangas
[2016-02-02T08:30:14.503] debug:  post user: couldn't get a uid for user jackmamc
[2016-02-02T08:30:14.505] debug:  post user: couldn't get a uid for user jacksjc1
[2016-02-02T08:30:14.507] debug:  post user: couldn't get a uid for user jaina4
[2016-02-02T08:30:14.521] debug:  post user: couldn't get a uid for user jonesbl2
[2016-02-02T08:30:14.524] debug:  post user: couldn't get a uid for user jovanon
[2016-02-02T08:30:14.526] debug:  post user: couldn't get a uid for user juarezaj
[2016-02-02T08:30:14.538] debug:  post user: couldn't get a uid for user kazmiek
[2016-02-02T08:30:14.546] debug:  post user: couldn't get a uid for user kemplepa
[2016-02-02T08:30:14.560] debug:  post user: couldn't get a uid for user kimp4
[2016-02-02T08:30:14.568] debug:  post user: couldn't get a uid for user kondoj
[2016-02-02T08:30:14.580] debug:  post user: couldn't get a uid for user kruegej
[2016-02-02T08:30:14.586] debug:  post user: couldn't get a uid for user kumara4
[2016-02-02T08:30:14.592] debug:  post user: couldn't get a uid for user kurelalr
[2016-02-02T08:30:14.599] debug:  post user: couldn't get a uid for user lanseram
[2016-02-02T08:30:14.604] debug:  post user: couldn't get a uid for user leblant
[2016-02-02T08:30:14.616] debug:  post user: couldn't get a uid for user lentscaj
[2016-02-02T08:30:14.636] debug:  post user: couldn't get a uid for user linc14
[2016-02-02T08:30:14.640] debug:  post user: couldn't get a uid for user linders
[2016-02-02T08:30:14.659] debug:  post user: couldn't get a uid for user liuy40
[2016-02-02T08:30:14.661] debug:  post user: couldn't get a uid for user lonergzr
[2016-02-02T08:30:14.666] debug:  post user: couldn't get a uid for user loveletd
[2016-02-02T08:30:14.668] debug:  post user: couldn't get a uid for user loweew
[2016-02-02T08:30:14.670] debug:  post user: couldn't get a uid for user lsslp
[2016-02-02T08:30:14.683] debug:  post user: couldn't get a uid for user mackml
[2016-02-02T08:30:14.685] debug:  post user: couldn't get a uid for user mah4
[2016-02-02T08:30:14.689] debug:  post user: couldn't get a uid for user mahatmnn
[2016-02-02T08:30:14.693] debug:  post user: couldn't get a uid for user maksymjt
[2016-02-02T08:30:14.706] debug:  post user: couldn't get a uid for user martinkp
[2016-02-02T08:30:14.717] debug:  post user: couldn't get a uid for user mayi
[2016-02-02T08:30:14.727] debug:  post user: couldn't get a uid for user mckinlet
[2016-02-02T08:30:14.729] debug:  post user: couldn't get a uid for user mdsallmi
[2016-02-02T08:30:14.755] debug:  post user: couldn't get a uid for user montalrj
[2016-02-02T08:30:14.763] debug:  post user: couldn't get a uid for user moraleca
[2016-02-02T08:30:14.780] debug:  post user: couldn't get a uid for user mueller
[2016-02-02T08:30:14.782] debug:  post user: couldn't get a uid for user muirmc
[2016-02-02T08:30:14.785] debug:  post user: couldn't get a uid for user mullinjg
[2016-02-02T08:30:14.810] debug:  post user: couldn't get a uid for user nidiffar
[2016-02-02T08:30:14.815] debug:  post user: couldn't get a uid for user ninomit
[2016-02-02T08:30:14.827] debug:  post user: couldn't get a uid for user ockenheb
[2016-02-02T08:30:14.834] debug:  post user: couldn't get a uid for user oliverkh
[2016-02-02T08:30:14.843] debug:  post user: couldn't get a uid for user osherojh
[2016-02-02T08:30:14.860] debug:  post user: couldn't get a uid for user parvatp
[2016-02-02T08:30:14.864] debug:  post user: couldn't get a uid for user pattans
[2016-02-02T08:30:14.875] debug:  post user: couldn't get a uid for user phillins
[2016-02-02T08:30:14.882] debug:  post user: couldn't get a uid for user powella5
[2016-02-02T08:30:14.884] debug:  post user: couldn't get a uid for user prokopjw
[2016-02-02T08:30:14.885] debug:  post user: couldn't get a uid for user purcelba
[2016-02-02T08:30:14.890] debug:  post user: couldn't get a uid for user reecet
[2016-02-02T08:30:14.891] debug:  post user: couldn't get a uid for user rhoy
[2016-02-02T08:30:14.892] debug:  post user: couldn't get a uid for user richasa1
[2016-02-02T08:30:14.899] debug:  post user: couldn't get a uid for user rowlanhg
[2016-02-02T08:30:14.902] debug:  post user: couldn't get a uid for user salimagj
[2016-02-02T08:30:14.912] debug:  post user: couldn't get a uid for user scheren1
[2016-02-02T08:30:14.916] debug:  post user: couldn't get a uid for user schnurja
[2016-02-02T08:30:14.918] debug:  post user: couldn't get a uid for user scurrac
[2016-02-02T08:30:14.919] debug:  post user: couldn't get a uid for user searfoam
[2016-02-02T08:30:14.920] debug:  post user: couldn't get a uid for user set
[2016-02-02T08:30:14.922] debug:  post user: couldn't get a uid for user shahidz
[2016-02-02T08:30:14.933] debug:  post user: couldn't get a uid for user siemanjk
[2016-02-02T08:30:14.934] debug:  post user: couldn't get a uid for user silvesm2
[2016-02-02T08:30:14.935] debug:  post user: couldn't get a uid for user simondm1
[2016-02-02T08:30:14.944] debug:  post user: couldn't get a uid for user sobotars
[2016-02-02T08:30:14.948] debug:  post user: couldn't get a uid for user spodemc1
[2016-02-02T08:30:14.967] debug:  post user: couldn't get a uid for user stergii
[2016-02-02T08:30:14.969] debug:  post user: couldn't get a uid for user stewarc8
[2016-02-02T08:30:14.973] debug:  post user: couldn't get a uid for user stumvom
[2016-02-02T08:30:14.974] debug:  post user: couldn't get a uid for user sundbyrt
[2016-02-02T08:30:14.976] debug:  post user: couldn't get a uid for user takizay
[2016-02-02T08:30:14.984] debug:  post user: couldn't get a uid for user theisgsj
[2016-02-02T08:30:14.985] debug:  post user: couldn't get a uid for user thelena
[2016-02-02T08:30:14.987] debug:  post user: couldn't get a uid for user thompsmw
[2016-02-02T08:30:14.991] debug:  post user: couldn't get a uid for user trenchal
[2016-02-02T08:30:14.994] debug:  post user: couldn't get a uid for user trujilca
[2016-02-02T08:30:14.996] debug:  post user: couldn't get a uid for user uberkr
[2016-02-02T08:30:17.020] debug:  post user: couldn't get a uid for user wangav
[2016-02-02T08:30:17.024] debug:  post user: couldn't get a uid for user wangl8
[2016-02-02T08:30:17.025] debug:  post user: couldn't get a uid for user wangq4
[2016-02-02T08:30:17.029] debug:  post user: couldn't get a uid for user wangy46
[2016-02-02T08:30:17.031] debug:  post user: couldn't get a uid for user weinstjj
[2016-02-02T08:30:17.037] debug:  post user: couldn't get a uid for user wildead
[2016-02-02T08:30:17.040] debug:  post user: couldn't get a uid for user willisjr
[2016-02-02T08:30:17.042] debug:  post user: couldn't get a uid for user wolfkm2
[2016-02-02T08:30:17.047] debug:  post user: couldn't get a uid for user wyszynkk
[2016-02-02T08:30:17.051] debug:  post user: couldn't get a uid for user yamagum1
[2016-02-02T08:30:17.052] debug:  post user: couldn't get a uid for user yangh7
[2016-02-02T08:30:17.062] debug:  post user: couldn't get a uid for user yvernabc
[2016-02-02T08:30:17.064] debug:  post user: couldn't get a uid for user zandbeb
[2016-02-02T08:30:17.065] debug:  post user: couldn't get a uid for user zellmat
[2016-02-02T08:30:17.067] debug:  post user: couldn't get a uid for user zhangb5
[2016-02-02T08:30:17.070] debug:  post user: couldn't get a uid for user zhangy39
[2016-02-02T08:30:17.075] debug:  post user: couldn't get a uid for user zhanx3
[2016-02-02T08:30:17.076] debug:  post user: couldn't get a uid for user zhany10
[2016-02-02T08:30:17.077] debug:  post user: couldn't get a uid for user zhaoj10
[2016-02-02T08:30:17.081] debug:  post user: couldn't get a uid for user zhouh4
[2016-02-02T08:30:17.085] debug:  post user: couldn't get a uid for user zonair
[2016-02-02T08:30:17.086] slurmdbd version 15.08.7 started
[2016-02-02T08:30:17.087] debug:  Reading slurm.conf file: /usr/scheduler/slurm-15.08.7/etc/slurm.conf
[2016-02-02T08:30:17.118] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59229 IP:10.0.0.49 CONN:7
[2016-02-02T08:30:28.094] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:9
[2016-02-02T08:30:30.714] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:8
[2016-02-02T08:35:30.880] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:12
[2016-02-02T08:40:27.383] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:8
[2016-02-02T08:40:30.355] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:9
[2016-02-02T08:45:30.828] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:12
[2016-02-02T08:50:24.546] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:176870 IP:10.0.20.240 CONN:8
[2016-02-02T08:50:28.217] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:9
[2016-02-02T08:50:30.342] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:8
[2016-02-02T08:55:30.793] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:12
[2016-02-02T08:56:17.640] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:176870 IP:10.0.23.252 CONN:8
[2016-02-02T08:56:31.652] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:176870 IP:10.0.23.252 CONN:9
[2016-02-02T08:56:46.397] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:176870 IP:10.0.23.252 CONN:8
[2016-02-02T08:56:56.859] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:176870 IP:10.0.23.252 CONN:9
[2016-02-02T08:57:12.095] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:176870 IP:10.0.23.252 CONN:8
[2016-02-02T08:57:23.546] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:176870 IP:10.0.23.252 CONN:9
[2016-02-02T08:57:37.275] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:176870 IP:10.0.23.252 CONN:8
[2016-02-02T08:58:03.088] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:176870 IP:10.0.23.252 CONN:9
[2016-02-02T08:58:30.975] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T08:58:31.105] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T08:58:31.225] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T08:58:57.588] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T08:58:57.717] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T08:58:57.843] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T08:59:20.587] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T08:59:20.717] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T08:59:20.845] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T08:59:44.807] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T08:59:44.936] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T08:59:45.063] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T09:00:15.113] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T09:00:15.242] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T09:00:15.367] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T09:00:23.353] debug:  Need to reroll usage from Tue Feb 02 08:59:51 2016 Job 7002630 from accre became eligible then and we are just now hearing about it.
[2016-02-02T09:00:27.582] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:9
[2016-02-02T09:00:30.323] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:8
[2016-02-02T09:00:38.962] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:12
[2016-02-02T09:00:39.097] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T09:00:39.249] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T09:00:57.023] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T09:00:57.128] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T09:00:57.247] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T09:01:10.854] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T09:01:10.956] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T09:01:11.058] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T09:01:24.791] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T09:01:24.893] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T09:01:24.994] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T09:01:37.386] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T09:01:37.489] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T09:01:37.591] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T09:01:49.477] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T09:01:49.581] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T09:01:49.683] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T09:02:00.676] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T09:02:00.779] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T09:02:00.880] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T09:02:11.771] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T09:02:11.874] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T09:02:11.975] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T09:02:30.955] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T09:02:31.059] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T09:02:31.169] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T09:02:43.688] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T09:02:43.792] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T09:02:43.894] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T09:02:55.924] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T09:02:56.028] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T09:02:56.130] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T09:05:30.875] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:8
[2016-02-02T09:10:28.074] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:9
[2016-02-02T09:10:30.373] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:8
[2016-02-02T09:15:30.862] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:12
[2016-02-02T09:20:27.595] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:8
[2016-02-02T09:20:30.528] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:9
[2016-02-02T09:25:30.585] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:12
[2016-02-02T09:30:27.790] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:8
[2016-02-02T09:30:31.039] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:9
[2016-02-02T09:35:30.855] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:12
[2016-02-02T09:40:28.326] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:8
[2016-02-02T09:40:30.329] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:9
[2016-02-02T09:44:15.857] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:111664 IP:10.0.20.238 CONN:12
[2016-02-02T09:44:39.521] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:111664 IP:10.0.20.238 CONN:8
[2016-02-02T09:44:54.967] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:111664 IP:10.0.20.238 CONN:9
[2016-02-02T09:45:30.228] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:8
[2016-02-02T09:50:27.540] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:9
[2016-02-02T09:50:30.600] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:8
[2016-02-02T09:52:54.797] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:111664 IP:10.0.20.238 CONN:12
[2016-02-02T09:53:30.912] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:111664 IP:10.0.20.238 CONN:8
[2016-02-02T09:55:31.108] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:9
[2016-02-02T10:00:27.584] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:8
[2016-02-02T10:00:30.679] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:9
[2016-02-02T10:05:31.101] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:12
[2016-02-02T10:10:27.445] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:8
[2016-02-02T10:10:30.626] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:9
[2016-02-02T10:15:29.137] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:111664 IP:10.0.20.238 CONN:12
[2016-02-02T10:15:31.141] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:8
[2016-02-02T10:18:13.189] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:111664 IP:10.0.20.238 CONN:9
[2016-02-02T10:20:27.862] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:8
[2016-02-02T10:20:30.238] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:9
[2016-02-02T10:23:19.413] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:111664 IP:10.0.20.238 CONN:12
[2016-02-02T10:24:34.648] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.51 CONN:8
[2016-02-02T10:25:30.759] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:9
[2016-02-02T10:27:08.734] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:111664 IP:10.0.20.238 CONN:8
[2016-02-02T10:30:27.520] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:9
[2016-02-02T10:30:30.267] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:8
[2016-02-02T10:35:30.379] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:12
[2016-02-02T10:40:28.322] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:8
[2016-02-02T10:40:30.911] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:9
[2016-02-02T10:45:30.449] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:12
[2016-02-02T10:50:28.368] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:8
[2016-02-02T10:50:30.969] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:9
[2016-02-02T10:55:30.475] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:12
[2016-02-02T11:00:06.412] debug:  Need to reroll usage from Tue Feb 02 10:59:34 2016 Job 7004689 from accre became eligible then and we are just now hearing about it.
[2016-02-02T11:00:28.035] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:8
[2016-02-02T11:00:31.119] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:9
[2016-02-02T11:03:16.860] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:5007 IP:10.0.2.232 CONN:12
[2016-02-02T11:03:16.970] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:5007 IP:10.0.2.232 CONN:8
[2016-02-02T11:03:17.074] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:5007 IP:10.0.2.232 CONN:9
[2016-02-02T11:04:23.845] debug:  Need to reroll usage from Tue Feb 02 10:45:19 2016 Job 7004935 from accre was submitted then and we are just now hearing about it.
[2016-02-02T11:05:31.020] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T11:05:31.089] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:9
[2016-02-02T11:05:31.148] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:12
[2016-02-02T11:05:31.277] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T11:05:58.278] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:11
[2016-02-02T11:05:58.407] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T11:05:58.535] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T11:06:18.249] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T11:06:18.378] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T11:06:18.505] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T11:06:38.475] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T11:06:38.604] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T11:06:38.730] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T11:07:17.549] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T11:07:17.680] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T11:07:17.807] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T11:07:37.322] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T11:07:37.452] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T11:07:37.579] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T11:08:20.204] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T11:08:20.333] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T11:08:20.460] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T11:08:45.420] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T11:08:45.550] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T11:08:45.689] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T11:09:04.235] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T11:09:04.364] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T11:09:04.492] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T11:09:29.380] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T11:09:29.509] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T11:09:29.636] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T11:09:48.354] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T11:09:48.484] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T11:09:48.612] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T11:10:19.110] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T11:10:19.239] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T11:10:19.365] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T11:10:28.492] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:8
[2016-02-02T11:10:30.586] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:9
[2016-02-02T11:10:37.506] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:12
[2016-02-02T11:10:37.641] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T11:10:37.768] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T11:11:17.722] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T11:11:17.854] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T11:11:17.984] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T11:11:55.885] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T11:11:56.017] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T11:11:56.145] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T11:12:16.020] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T11:12:16.149] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T11:12:16.277] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T11:12:35.766] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T11:12:35.895] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T11:12:36.021] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T11:12:54.693] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T11:12:54.816] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T11:12:54.960] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T11:13:12.787] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T11:13:12.916] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T11:13:13.133] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T11:13:31.338] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T11:13:31.467] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T11:13:31.595] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T11:14:23.048] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T11:14:23.175] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T11:14:23.304] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T11:15:13.641] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T11:15:13.770] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T11:15:13.895] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T11:15:31.107] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:9
[2016-02-02T11:15:33.729] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:5007 IP:10.0.2.232 CONN:8
[2016-02-02T11:15:33.834] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:5007 IP:10.0.2.232 CONN:9
[2016-02-02T11:15:33.951] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:5007 IP:10.0.2.232 CONN:8
[2016-02-02T11:16:45.189] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T11:16:45.318] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T11:16:45.445] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T11:20:27.684] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:8
[2016-02-02T11:20:31.222] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:9
[2016-02-02T11:25:30.727] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:12
[2016-02-02T11:27:49.483] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:5007 IP:10.0.2.232 CONN:8
[2016-02-02T11:27:49.589] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:5007 IP:10.0.2.232 CONN:9
[2016-02-02T11:27:49.690] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:5007 IP:10.0.2.232 CONN:8
[2016-02-02T11:28:26.831] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:5007 IP:10.0.2.232 CONN:9
[2016-02-02T11:28:26.928] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:5007 IP:10.0.2.232 CONN:8
[2016-02-02T11:28:27.023] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:5007 IP:10.0.2.232 CONN:9
[2016-02-02T11:30:27.613] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:8
[2016-02-02T11:30:31.230] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:9
[2016-02-02T11:35:31.223] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:12
[2016-02-02T11:40:28.182] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:8
[2016-02-02T11:40:30.723] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:9
[2016-02-02T11:45:31.204] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:12
[2016-02-02T11:50:28.176] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:8
[2016-02-02T11:50:30.599] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:9
[2016-02-02T11:55:31.152] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:12
[2016-02-02T12:00:15.488] debug:  Need to reroll usage from Tue Feb 02 11:59:41 2016 Job 7007267 from accre started then and we are just now hearing about it.
[2016-02-02T12:00:27.971] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:8
[2016-02-02T12:00:30.754] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:9
[2016-02-02T12:00:51.251] debug:  Need to reroll usage from Tue Feb 02 11:41:10 2016 Job 7006380 from accre became eligible then and we are just now hearing about it.
[2016-02-02T12:05:30.775] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:12
[2016-02-02T12:06:12.987] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.4.241 CONN:8
[2016-02-02T12:07:14.937] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.4.241 CONN:9
[2016-02-02T12:10:27.698] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:8
[2016-02-02T12:10:31.253] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:9
[2016-02-02T12:15:30.727] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:12
[2016-02-02T12:20:27.573] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:8
[2016-02-02T12:20:30.573] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:9
[2016-02-02T12:25:31.076] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:12
[2016-02-02T12:30:28.630] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:8
[2016-02-02T12:30:30.567] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:9
[2016-02-02T12:35:30.488] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:12
[2016-02-02T12:40:27.684] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:8
[2016-02-02T12:40:31.043] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:9
[2016-02-02T12:45:09.630] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:156409 IP:10.0.20.241 CONN:12
[2016-02-02T12:45:30.528] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:8
[2016-02-02T12:50:27.767] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:9
[2016-02-02T12:50:30.919] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:8
[2016-02-02T12:55:30.473] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:12
[2016-02-02T13:00:01.413] debug:  Need to reroll usage from Tue Feb 02 12:59:28 2016 Job 7007596 from accre became eligible then and we are just now hearing about it.
[2016-02-02T13:00:27.679] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:8
[2016-02-02T13:00:31.011] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:9
[2016-02-02T13:02:27.517] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:12
[2016-02-02T13:02:27.620] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T13:02:27.723] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T13:03:14.713] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T13:03:14.816] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T13:03:14.918] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T13:03:42.167] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T13:03:42.269] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T13:03:42.372] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T13:04:00.093] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T13:04:00.197] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T13:04:00.299] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T13:04:18.126] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T13:04:18.229] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T13:04:18.331] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T13:04:33.087] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T13:04:33.190] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T13:04:33.291] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T13:04:45.835] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T13:04:45.939] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T13:04:46.041] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T13:05:31.177] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:8
[2016-02-02T13:06:16.784] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T13:06:16.918] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T13:06:17.045] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T13:06:35.444] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T13:06:35.565] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T13:06:35.839] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T13:06:59.781] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:19692 IP:10.0.20.242 CONN:9
[2016-02-02T13:07:00.013] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:19692 IP:10.0.20.242 CONN:8
[2016-02-02T13:07:07.150] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T13:07:07.283] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T13:07:07.412] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T13:07:25.672] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T13:07:25.805] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T13:07:25.936] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T13:07:49.049] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T13:07:49.181] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T13:07:49.308] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T13:08:25.073] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T13:08:25.205] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T13:08:25.329] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T13:08:43.242] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T13:08:43.374] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T13:08:43.504] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T13:09:06.978] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T13:09:07.106] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T13:09:07.224] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T13:10:28.201] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:9
[2016-02-02T13:10:30.713] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:8
[2016-02-02T13:10:55.178] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:12
[2016-02-02T13:10:55.310] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T13:10:55.439] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T13:12:03.008] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T13:12:03.141] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T13:12:03.270] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T13:12:19.852] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T13:12:19.986] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T13:12:20.119] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T13:12:44.141] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T13:12:44.271] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T13:12:44.395] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T13:13:01.969] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T13:13:02.100] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T13:13:02.219] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T13:13:19.793] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T13:13:19.923] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T13:13:20.041] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T13:13:53.760] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T13:13:53.892] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T13:13:54.012] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T13:14:12.867] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T13:14:12.999] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T13:14:13.124] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T13:14:31.559] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T13:14:31.692] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T13:14:31.819] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T13:14:49.748] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T13:14:49.881] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T13:14:50.010] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T13:15:21.269] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T13:15:21.400] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T13:15:21.526] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T13:15:31.248] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:8
[2016-02-02T13:17:48.394] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T13:17:48.525] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T13:17:48.655] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T13:20:27.628] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:8
[2016-02-02T13:20:30.483] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:9
[2016-02-02T13:25:30.992] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:12
[2016-02-02T13:28:09.333] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:112888 IP:10.0.20.241 CONN:8
[2016-02-02T13:28:16.255] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:112888 IP:10.0.20.241 CONN:9
[2016-02-02T13:28:16.449] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:112888 IP:10.0.20.241 CONN:8
[2016-02-02T13:30:27.804] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:9
[2016-02-02T13:30:30.516] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:8
[2016-02-02T13:35:30.579] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:12
[2016-02-02T13:38:11.043] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:112888 IP:10.0.23.251 CONN:8
[2016-02-02T13:38:12.720] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:112888 IP:10.0.23.251 CONN:9
[2016-02-02T13:38:12.914] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:112888 IP:10.0.23.251 CONN:8
[2016-02-02T13:40:28.297] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:9
[2016-02-02T13:40:31.103] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:8
[2016-02-02T13:41:50.705] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:19692 IP:10.0.20.242 CONN:12
[2016-02-02T13:41:51.061] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:19692 IP:10.0.20.242 CONN:8
[2016-02-02T13:45:30.595] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:9
[2016-02-02T13:50:27.915] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:8
[2016-02-02T13:50:30.823] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:9
[2016-02-02T13:55:31.319] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:12
[2016-02-02T13:56:46.039] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:19692 IP:10.0.20.242 CONN:8
[2016-02-02T13:56:46.290] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:19692 IP:10.0.20.242 CONN:9
[2016-02-02T14:00:27.831] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:8
[2016-02-02T14:00:30.847] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:9
[2016-02-02T14:05:30.981] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:12
[2016-02-02T14:10:28.601] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:8
[2016-02-02T14:10:30.567] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:9
[2016-02-02T14:12:00.668] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:5007 IP:10.0.2.232 CONN:12
[2016-02-02T14:13:15.707] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:5007 IP:10.0.2.232 CONN:8
[2016-02-02T14:14:22.726] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:5007 IP:10.0.2.232 CONN:9
[2016-02-02T14:15:31.083] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:8
[2016-02-02T14:20:28.350] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:9
[2016-02-02T14:20:31.140] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:8
[2016-02-02T14:25:30.673] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:12
[2016-02-02T14:30:28.156] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:8
[2016-02-02T14:30:31.181] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:9
[2016-02-02T14:35:31.251] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:12
[2016-02-02T14:40:28.748] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:8
[2016-02-02T14:40:30.770] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:9
[2016-02-02T14:45:31.296] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:12
[2016-02-02T14:50:27.953] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:8
[2016-02-02T14:50:30.923] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:9
[2016-02-02T14:55:31.444] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:12
[2016-02-02T15:00:28.084] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:8
[2016-02-02T15:00:30.966] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:9
[2016-02-02T15:03:04.679] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:5007 IP:10.0.2.232 CONN:12
[2016-02-02T15:03:04.785] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:5007 IP:10.0.2.232 CONN:8
[2016-02-02T15:03:04.889] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:5007 IP:10.0.2.232 CONN:9
[2016-02-02T15:04:36.428] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T15:04:36.532] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T15:04:36.634] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T15:04:49.235] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T15:04:49.338] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T15:04:49.441] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T15:05:00.811] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T15:05:00.914] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T15:05:01.017] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T15:05:30.759] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:9
[2016-02-02T15:06:06.202] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T15:06:06.306] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T15:06:06.409] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T15:06:18.893] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T15:06:18.996] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T15:06:19.099] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T15:06:26.439] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T15:06:26.570] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T15:06:26.687] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T15:06:31.920] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T15:06:32.023] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T15:06:32.126] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T15:06:43.604] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T15:06:43.707] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T15:06:43.809] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T15:06:55.005] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T15:06:55.109] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T15:06:55.211] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T15:07:06.027] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T15:07:06.132] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T15:07:06.234] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T15:07:17.290] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T15:07:17.393] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T15:07:17.495] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T15:07:17.786] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T15:07:17.904] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T15:07:18.032] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T15:07:29.422] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T15:07:29.528] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T15:07:29.633] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T15:07:40.428] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T15:07:40.531] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T15:07:40.634] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T15:07:51.712] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T15:07:51.815] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T15:07:51.918] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T15:07:55.280] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T15:07:55.412] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T15:07:55.541] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T15:08:04.215] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T15:08:04.319] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T15:08:04.420] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T15:08:15.134] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T15:08:15.237] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T15:08:15.339] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T15:08:26.198] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T15:08:26.331] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T15:08:26.462] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T15:08:27.004] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T15:08:27.107] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T15:08:27.211] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T15:08:38.065] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T15:08:38.169] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T15:08:38.271] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T15:08:49.358] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T15:08:49.461] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:9
[2016-02-02T15:08:49.563] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:500 IP:10.0.2.229 CONN:8
[2016-02-02T15:09:45.738] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T15:09:45.870] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T15:09:46.000] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T15:10:28.759] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:8
[2016-02-02T15:10:31.288] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:9
[2016-02-02T15:10:36.058] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:12
[2016-02-02T15:10:36.189] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T15:10:36.319] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T15:11:55.353] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T15:11:55.485] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T15:11:55.614] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T15:12:38.068] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T15:12:38.199] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T15:12:38.326] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T15:13:46.209] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T15:13:46.339] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T15:13:46.468] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T15:14:24.080] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T15:14:24.211] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:8
[2016-02-02T15:14:24.340] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:59223 IP:10.0.2.231 CONN:9
[2016-02-02T15:15:30.794] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:8
[2016-02-02T15:20:28.029] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:9
[2016-02-02T15:20:30.774] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:8
[2016-02-02T15:25:31.357] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:12
[2016-02-02T15:30:28.400] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:8
[2016-02-02T15:30:30.896] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:9
[2016-02-02T15:35:30.703] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:12
[2016-02-02T15:40:28.155] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:8
[2016-02-02T15:40:31.244] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:9
[2016-02-02T15:45:30.761] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:12
[2016-02-02T15:50:28.286] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:1003 IP:10.0.22.51 CONN:8
[2016-02-02T15:50:30.709] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:9
[2016-02-02T15:55:31.277] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:0 IP:10.0.0.49 CONN:12
[2016-02-02T15:55:52.541] debug:  DBD_INIT: CLUSTER:accre VERSION:7424 UID:162009 IP:10.0.20.242 CONN:8
[2016-02-02T16:00:04.698] debug:  Need to reroll usage from Tue Feb 02 15:59:32 2016 Job 7012284 from accre became eligible then and we are just now hearing about it.

Comment 14 Charles Johnson 2016-02-02 08:06:26 MST

"Do you have logrotate or something else running on the slurmdbd host that may
have caused this?"

No. We rotate by size and the last rotation was several months ago.

"What OS is slurmdbd running on? I do recall seeing some issues with systemd,
and we did change the example service scripts back in November."

CentOS 6.x

"Did anything else change with that update? Did you switch OS versions at the
same time, or change how Slurm is being launched?"

Nothing changed. We do our best to completely isolate changes to slurm for just this reason. Too many rabbits to chase otherwise!

This morning I did set slurmdb to gather 1 second's worth of data before a commit to mariaDB.

Charles

Comment 15 Tim Wickberg 2016-02-02 08:12:41 MST

I'd really love to know what command is running as root every five minutes on 10.0.0.49 - can you share what's in that cron job?

Comment 16 Will French 2016-02-02 10:00:52 MST

(In reply to Tim Wickberg from comment #15)
> I'd really love to know what command is running as root every five minutes
> on 10.0.0.49 - can you share what's in that cron job?

root@slurmsched1:~# crontab -l | grep "*/5"
*/5 * * * * /home/rrdtool/bin/SlurmUpdate.pl
*/5 * * * * /home/rrdtool/bin/GridSlurmUpdate.pl

We have a couple of perl scripts running that generate plots/images summarizing cluster utilization and get copied to a web-accessible location.

SlurmUpdate.pl is running squeue once and "scontrol show nodes" once.

GridSlurmUpdate.pl is running squeue once and "sacctmgr show associations" once.

The remainder of these scripts are dedicated to parsing the output from these SLURM commands. These have existed since before we started using SLURM a year ago (we had to adapt them to parse SLURM command output), although I am noticing now that some of the fields in "sacctmgr show associations" have changed in version 15 and that the second script needs to be updated to account for that.

Comment 17 Charles Johnson 2016-02-02 22:33:28 MST

I have attached the two perl scripts, but they seem pretty innocuous, since they parse either showq-wrapper, scontrol or squeue output. They could be run on a different machine if needed.

Charles

Comment 18 Charles Johnson 2016-02-02 22:35:27 MST

Created attachment 2681 [details]
SlurmUpdate.pl (perl script)

Comment 19 Charles Johnson 2016-02-02 22:36:42 MST

Created attachment 2682 [details]
GridSlurmUpdate.pl (perl script)

Comment 20 Tim Wickberg 2016-02-11 10:03:34 MST

I've realized I skipped right past our normal MySQL tuning questions; if you haven't tuned MySQL's innodb_buffer_pool_size that may explain what appear to be sporadic delays in processing for slurmdbd.

If you haven't adjusted it, it's likely set quite small by default. If you can set it to (assuming the machine has sufficient RAM) 2GB or higher that should rule that out as a potential source of problems.

https://dev.mysql.com/doc/refman/5.5/en/innodb-buffer-pool.html

Comment 21 Tim Wickberg 2016-03-04 01:19:00 MST

Checking back in - did adjusting innodb_buffer_pool_size clear this up, or are you still seeing problems?

- Tim

Comment 22 Will French 2016-03-04 08:34:39 MST

(In reply to Tim Wickberg from comment #21)
> Checking back in - did adjusting innodb_buffer_pool_size clear this up, or
> are you still seeing problems?

Hey Tim,

Charles has been out on vacation for a few weeks but he'll be back on Monday. I think he did end up adjusting this value but just want to verify with him first. 

In either case, the problem seems to have subsided. 

We'll check back in next week.

Will

Comment 23 Tim Wickberg 2016-03-15 08:11:38 MDT

*ping*.

I'm hoping things are working well for you, and if you don't have anything further I'll go ahead and close this out.

cheers,
- Tim

(In reply to Will French from comment #22)
> (In reply to Tim Wickberg from comment #21)
> > Checking back in - did adjusting innodb_buffer_pool_size clear this up, or
> > are you still seeing problems?
> 
> Hey Tim,
> 
> Charles has been out on vacation for a few weeks but he'll be back on
> Monday. I think he did end up adjusting this value but just want to verify
> with him first. 
> 
> In either case, the problem seems to have subsided. 
> 
> We'll check back in next week.
> 
> Will

Comment 24 Charles Johnson 2016-03-15 08:21:53 MDT

On 03/15/2016 04:11 PM, bugs@schedmd.com wrote:
>
> *Comment # 23 <https://bugs.schedmd.com/show_bug.cgi?id=2395#c23> on 
> bug 2395 <https://bugs.schedmd.com/show_bug.cgi?id=2395> from Tim 
> Wickberg <mailto:tim@schedmd.com> *
> *ping*.
>
> I'm hoping things are working well for you, and if you don't have anything
> further I'll go ahead and close this out.
>
> cheers,
> - Tim
>
> (In reply to Will French fromcomment #22 <show_bug.cgi?id=2395#c22>)
> > (In reply to Tim Wickberg fromcomment #21 <show_bug.cgi?id=2395#c21>) > > Checking back in - did 
> adjusting innodb_buffer_pool_size clear this up, or > > are you still 
> seeing problems? > > Hey Tim, > > Charles has been out on vacation for 
> a few weeks but he'll be back on > Monday. I think he did end up 
> adjusting this value but just want to verify > with him first. > > In 
> either case, the problem seems to have subsided. > > We'll check back 
> in next week. > > Will
> ------------------------------------------------------------------------
> You are receiving this mail because:
>
>   * You reported the bug.
>

I think I should have replied. All looks good. Please, close the ticket 
and sorry about the delay to get back to you.

Charles

Comment 25 Tim Wickberg 2016-03-15 08:22:50 MDT

No problem, just wanted to make sure I hadn't overlooked anything.

- Tim

Comment 26 Charles Johnson 2016-03-16 00:14:30 MDT

Tim, I have reopened this to give you a bit more information before it gets closed. I was precipitous in closing it. Sorry about that!

slurmdb and mariaDB have now been up without interruption since March 4th. This has given mariaDB sufficient time to collect data about how it is being used in our environment by slurm.

First, some hardware info: the server machine is a dual quad, with 128GB ram. Linux is using 125GB, and of that 109GB is cached. That seems to indicate that mariaDB is now pretty much RAM resident.

For mariahDB we now have these factors in play:

[mysqld]
innodb_buffer_pool_size=64G
query_cache_type=1
query_cache_limit=128M
query_cache_size=128M

tmp_table_size=64M
max_heap_table_size=64M

innodb_log_buffer_size=32M
# https://bugs.schedmd.com/show_bug.cgi?id=2509
innodb_log_file_size=64M
innodb_lock_wait_timeout=900
# https://bugs.schedmd.com/show_bug.cgi?id=2509

So, we give half of RAM for the buffer_pool_size. It seems happy with that. When I query it for specific information mariaDB reports the following:

You have 52 out of 12706636 that take longer than 10.000000 sec. to complete
That seems OK to me. Way too few to worry about.

Current max_connections = 151
Current threads_connected = 3
Historic max_used_connections = 5
The number of used connections is 3% of the configured maximum.
We could probably lower our connection limit to gain back a bit of RAM.

Max Memory Ever Allocated : 64.30 G
Configured Max Per-thread Buffers : 419 M
Configured Max Global Buffers : 64.28 G
Configured Max Memory Limit : 64.69 G
Physical Memory : 125.99 G
mariaDB seems happy with this RAM allocation approach.

Current query_cache_size = 128 M
Current query_cache_used = 1 M
Current query_cache_limit = 128 M
Current Query cache Memory fill ratio = 1.06 %
Current query_cache_min_res_unit = 4 K
We could probably lower the cache size, since we seem to be grossly under-filling it.

Current join_buffer_size = 132.00 K
We have had 8 queries where a join could not use an index properly.
We have had 4 joins without keys that check for key usage after each row
We should probably enable "log-queries-not-using-indexes" and look for non indexed joins in the slow query log. We could report those to you, though you likely already know who the culprit are. Since we are unable to optimize your queries we may want to increase our join_buffer_size to accommodate larger joins in one pass. We could consider moving to 256K, or even 500K.

Current max_heap_table_size = 64 M
Current tmp_table_size = 64 M
Of 10918 temp tables, 45% were created on disk
Perhaps we should increase our tmp_table_size and/or max_heap_table_size
to reduce the number of disk-based temporary tables.

Other metrics I query for such as Table Caching (all tables currently cached), Sort Buffer Usage (we allocate a 2MB buffer and max historical usage has been 256K) etc., etc., all seem fine.

At the next down time we might consider running OPTIMIZE TABLE to defragment tables for better performance.

If, after reading through this, you have any comments, they would be welcome. Otherwise, go ahead and re-close this ticket.

Thanks,

Charles

BTW, if you haven't been to New Zealand for their summer (our winter) go!

Comment 27 Tim Wickberg 2016-03-16 06:04:41 MDT

(In reply to Charles Johnson from comment #26)
> Tim, I have reopened this to give you a bit more information before it gets
> closed. I was precipitous in closing it. Sorry about that!

No problem at all.

> slurmdb and mariaDB have now been up without interruption since March 4th.
> This has given mariaDB sufficient time to collect data about how it is being
> used in our environment by slurm.

Glad to hear that.

(*snip*)

Your analysis all looks good - it sounds like you have the tuning well in hand now, and everything we usually recommend has been done. I'll point to that comment in the future for sites asking how to approach MySQL tuning - the defaults most distributions ship with seem to be done for small web-apps, and don't handle the torrent of job data slurm can generate, but figuring out what and how to improve it is tricky if you're not familiar.

> Current max_heap_table_size = 64 M
> Current tmp_table_size = 64 M
> Of 10918 temp tables, 45% were created on disk
> Perhaps we should increase our tmp_table_size and/or max_heap_table_size
> to reduce the number of disk-based temporary tables. 

That sounds like it may be worth increasing, I can't imagine this is helping. But that's likely only for longer-running queries like excessively large sacct queries, and maybe the hourly/daily/monthly rollups, which aren't likely to suffer too much by this.
 
> At the next down time we might consider running OPTIMIZE TABLE to defragment
> tables for better performance.
> 
> If, after reading through this, you have any comments, they would be
> welcome. Otherwise, go ahead and re-close this ticket.

I can multitask. Marking closed again.

> BTW, if you haven't been to New Zealand for their summer (our winter) go!

I've been looking for an excuse to get down to that side of the globe... if you have any colleagues working down around Auckland or Australia that may need Slurm training let us know, I've been trying to get Jacob to find an excuse to send me down there. :)

cheers,
- Tim