Ticket 7906

Created attachment 11892 [details]
gres.conf

Created attachment 11893 [details]
slurmdbd.conf

Created attachment 11894 [details]
database.output

Hi D - 

"sacctmgr list cluster" should list the connection which it does not in your case.

For example from my cluster you see the registration with an entry with ControlHost,ControlPort.


$ sacctmgr list clusters format=Cluster,ControlHost,ControlPort,RPC,Share,QOS
   Cluster     ControlHost  ControlPort   RPC     Share                  QOS 
---------- --------------- ------------ ----- --------- -------------------- 
   cluster       127.0.0.1         8817  8704         1               normal 




Please check that you are not using SELinux or a firewall which may be preventing connection upto the database.


Is "AccountingStorageHost=it-test.corp.ai2.in" the same box that runs slurmctld?

There is nothing blocking the call as far as I can tell. I can see the traffic and the response in tcpdump (from both it-test and the dbhost) when making both sacct and sacctmgr calls, and I can log into the db from a mysql client.

darrellp@it-test ~ $ mysql -h util02 -u slurm -p 
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 10
Server version: 5.7.26-0ubuntu0.16.04.1 (Ubuntu)

Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> 


This is in addition to the fact that the sacctmgr add cluster allennlp altered the db. I do not believe that there is anything blocking traffic.  

As for the AccountstorageHost, that is correct, however this is using the FQDN, whereas other parts of the config are using the short name, so I changed this to the short name and restarted to see if that could be it (I have seen stranger things), but the same problem persists. 

Thanks
D

Would you please set debug2 and bounce the slurmctld and then gather the slurmctld.log for us to review?

You should see entries such as:

 debug:  slurmdbd: Sent PersistInit msg
or

debug2: slurm_connect failed: Connection refused
debug2: Error connecting slurm stream socket at 127.0.0.1:7920: Connection refused
error: slurmdbd: Sending PersistInit msg: Connection refused

Please make sure to change back to a lower loglevel when finished.

Well that is annoying. After adding the debugging flag and restarting, it suddenly decided to work. All I did was add these two lines: 

SlurmctldDebug=debug2
SlurmctldLogFile=/var/log/slurmctld.log

Now I have restarted everything in the past to no avail, but here is what I got just now.. 


darrellp@it-test /etc/slurm-llnl $ sudo vi slurm.conf 
darrellp@it-test /etc/slurm-llnl $ sudo systemctl restart slurmctld
darrellp@it-test /etc/slurm-llnl $ sudo systemctl restart slurmd
darrellp@it-test /etc/slurm-llnl $ sacct -a
       JobID    JobName  Partition    Account  AllocCPUS      State ExitCode 
------------ ---------- ---------- ---------- ---------- ---------- -------- 
darrellp@it-test /etc/slurm-llnl $ sacctmgr list cluster
   Cluster     ControlHost  ControlPort   RPC     Share GrpJobs       GrpTRES GrpSubmit MaxJobs       MaxTRES MaxSubmit     MaxWall                  QOS   Def QOS 
---------- --------------- ------------ ----- --------- ------- ------------- --------- ------- ------------- --------- ----------- -------------------- --------- 
  allennlp       127.0.0.1         6817  8704         1                                                                                           normal           
(failed reverse-i-search)`stun': sacctmgr list clu^Cer
darrellp@it-test /etc/slurm-llnl $ srun -w it-test -G 1 --pty bash -i 
darrellp@it-test /etc/slurm-llnl $ ls
gres.conf  slurm.conf  slurmdbd.conf
darrellp@it-test /etc/slurm-llnl $ exit
exit
darrellp@it-test /etc/slurm-llnl $ sacct -a
       JobID    JobName  Partition    Account  AllocCPUS      State ExitCode 
------------ ---------- ---------- ---------- ---------- ---------- -------- 
11                 bash   allennlp                     2  COMPLETED      0:0 



Is there any insight that you can derive from this? 

(attaching the debug log)

Created attachment 11908 [details]
slurmctld-debug.log

>Is there something obvious that I am missing here? Attaching my configs and db info  


Unfortunately I only have the logs from after the most recent restart so I can not comment further on what happened in the past. The logs I have now look completely normal. If restarts did not work in the past but do now then this may suggest some type of network communication issue between the two. Do you have a network admin that is pushing out security changes that might have affected this?

Nope. There have been no network changes in time of working this bug (that
team reports to me)

Cheers
D

On Thu, Oct 10, 2019 at 1:00 PM <bugs@schedmd.com> wrote:

> *Comment # 9 <https://bugs.schedmd.com/show_bug.cgi?id=7906#c9> on bug
> 7906 <https://bugs.schedmd.com/show_bug.cgi?id=7906> from Jason Booth
> <jbooth@schedmd.com> *
>
> >Is there something obvious that I am missing here? Attaching my configs and db info
>
>
> Unfortunately I only have the logs from after the most recent restart so I can
> not comment further on what happened in the past. The logs I have now look
> completely normal. If restarts did not work in the past but do now then this
> may suggest some type of network communication issue between the two. Do you
> have a network admin that is pushing out security changes that might have
> affected this?
>
> ------------------------------
> You are receiving this mail because:
>
>    - You reported the bug.
>
>

If you have the controller logs before the raise in debug level then I can review those to see if I can spot the disconnect, otherwise, we can move to close out this case.

We do but it was dumping to syslog so it is buried and likely incomplete. I think I have the config to go forward in our allennlp cluster and if I see it there, I will re-open. In the mean time, thank you Jason

Feel free to close this now. 

Cheers
D