Ticket 686 - slurm rpc:1008 : Communication connection failure
Summary: slurm rpc:1008 : Communication connection failure
Status: RESOLVED INFOGIVEN
Alias: None
Product: Slurm
Classification: Unclassified
Component: slurmctld (show other tickets)
Version: 2.6.2
Hardware: Linux Linux
: 3 - Medium Impact
Assignee: David Bigagli
QA Contact:
URL:
Depends on:
Blocks:
 
Reported: 2014-04-08 21:38 MDT by toru matsuoka
Modified: 2014-04-16 03:33 MDT (History)
1 user (show)

See Also:
Site: CRAY
Slinky Site: ---
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
Google sites: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
NoveTech Sites: ---
Nvidia HWinf-CS Sites: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Tzag Elita Sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed:
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---

Attachments
slurm.conf file (4.38 KB, text/plain)
2014-04-08 21:38 MDT, toru matsuoka 		Details
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this ticket.
Description toru matsuoka 2014-04-08 21:38:15 MDT 
Created attachment 739 [details]
slurm.conf file

Hello,Slurm support team !

I'm toru matsuoka in cray Japan Inc engineer.

A RPC 1008  Communication connection failure occurs of the Customer site whom I am taking charge of, and the trouble which a job stops have occurred frequently.
 
What has originated [ what ] and generated this trouble?
 
I attach a slurm.conf file. 

The contents of slurmctld.log are indicated below. 

This Trouble is occur at multiple nodes.

Please advice me about this trouble situation. 


Best Regards..
----------------------
Toru Matsuoka

Cray Japan Inc..
----------------------
Comment 1 Moe Jette 2014-04-09 03:14:09 MDT 
This error typically indicates 
1) some network problem
2) some slurmd daemons on some compute node are down or
3) the munged daemon on some compute node being down
If you are just starting the system and not all of the slurmd daemons are up, this error would be expected. It would also be expected if any of your compute nodes are down.

Your configuration looks fine.

Please look at the troubleshooting guide at the below site, especially the section on network problems.
http://slurm.schedmd.com/troubleshoot.html#network
Comment 2 Moe Jette 2014-04-15 08:47:54 MDT 
Were you able to identify the source of this problem?
Comment 3 toru matsuoka 2014-04-15 17:42:03 MDT 
Hello,Slurm Support team! 

thanks for always support.

The Cause of this trouble was network connection Problem.

At Customer site , Slurm integration was done.

Access of a network became impossible temporarily under the influence.

Now,Trouble status was became OK.

Please close this case.

Best Regards..
----------------------
Toru Matsuoka

Cray Japan Inc..
----------------------
Comment 4 Moe Jette 2014-04-16 03:33:58 MDT 
network problem at customer site resolved