Ticket 5401

Summary:	Nodes hang in completing state
Product:	Slurm	Reporter:	Nikolas Luke <niko.luke>
Component:	slurmctld	Assignee:	Jason Booth <jbooth>
Status:	RESOLVED CANNOTREPRODUCE	QA Contact:
Severity:	3 - Medium Impact
Priority:	---
Version:	17.11.7
Hardware:	Linux
OS:	Linux
Site:	Hessen	Slinky Site:	---
Alineos Sites:	---	Atos/Eviden Sites:	---
Confidential Site:	---	Coreweave sites:	---
Cray Sites:	---	DS9 clusters:	---
Google sites:	---	HPCnow Sites:	---
HPE Sites:	---	IBM Sites:	---
NOAA SIte:	---	NoveTech Sites:	---
Nvidia HWinf-CS Sites:	---	OCF Sites:	---
Recursion Pharma Sites:	---	SFW Sites:	---
SNIC sites:	---	Tzag Elita Sites:	---
Linux Distro:	---	Machine Name:
CLE Version:		Version Fixed:
Target Release:	---	DevPrio:	---
Emory-Cloud Sites:	---
Attachments:	Nodelist in completion state slurm.conf Logfile of one node (its-cs114) in completing state slurmctld.log grep its-cs114 ss -p Output of its-cs107 slurm.epilog Script

Description Nikolas Luke 2018-07-10 05:40:10 MDT

Some nodes often do not return out of the completing state. I have written a script which react after 15min of completing state of a node and tries

scontrol update NodeName=$NODENAME State=down Reason="comp"
sleep 10
scontrol update NodeName=$NODENAME State=idle

In most of the cases it works, but if not i have to restart the slurmctld, because a restart of the slurmd on the concerned nodes does not affect the completing state.

I use the default epilog script (named slurm.epilog) and cannot see it running, when I type

ps aux | grep -i slurm

Why does nodes hang in this state forever?

Comment 1 Nikolas Luke 2018-07-10 05:42:25 MDT

Created attachment 7255 [details]
Nodelist in completion state

Comment 2 Nikolas Luke 2018-07-10 05:46:26 MDT

Created attachment 7256 [details]
slurm.conf

Comment 3 Nikolas Luke 2018-07-10 05:54:47 MDT

Created attachment 7257 [details]
Logfile of one node (its-cs114) in completing state

Comment 5 Isaac Hartung 2018-07-11 00:16:43 MDT

Hi Nikolas,

I am looking into this and will keep you posted on any progress.  If there is any information in the slurmctld logs related to these nodes, that might be also be helpful.

Regards,

Isaac

Comment 6 Nikolas Luke 2018-07-11 02:55:43 MDT

Created attachment 7274 [details]
slurmctld.log grep its-cs114

Comment 7 Nikolas Luke 2018-07-11 03:04:30 MDT

I have added the slurmctld.log and grep about the same example node. You can see my attempts of set it down and up again. Sometimes there is a "Communication connection failure" for 10 minutes in the log. Even if there were sometimes network failures there is no need to stay in completion status forever and currently the network connection is ok.

Comment 8 Isaac Hartung 2018-07-31 09:04:54 MDT

Are the hanging nodes always the same nodes?  

Are there certain jobs that seem to trigger these situations more than others?

Comment 9 Nikolas Luke 2018-08-01 08:00:54 MDT

The problem occurs more often in a particular partition and a few individual nodes. The partition is with other partitions on the same ethernet and infiniband switch, so I think it's not the hardware or the connection.

Comment 10 Isaac Hartung 2018-08-20 17:33:50 MDT

Judging from your logs and slurm.conf, is the partition FB16?  And are the nodes that typically hang its-cs[107-118] or some subset of those?

Comment 11 Nikolas Luke 2018-08-21 09:17:28 MDT

Often the FB16 parcition is complete in comp-status, but sometimes only a few while the others are idle or allocated. Maybe they have all the same problem in this times, if they would all run the slurm.epilog.

Comment 12 Jason Booth 2018-08-28 11:22:26 MDT

Hi Nikolas,

 When the issue happens again would you please gather the output of "sdiag" and send this back to me?

Also, please gather the output of 'ss -p'. Does your site use a firewall on the node or head node and is SELinux enabled?

The messages that are being reported suggest that communication breaks down between the head node and the compute node and the job complete message never makes it back to the slurmctld. The slurmctld believes the job is in a completing state and will then re-send an instruction to kill the job over and over. If you can reproduce this consistently then I would also suggest raising the log level on one of the nodes to debug5 and send a job to this node. I would also be curious to know what actions you epilogue script is doing.

-Jason

Comment 13 Nikolas Luke 2018-09-07 02:58:39 MDT

Created attachment 7782 [details]
ss -p Output of its-cs107

Comment 14 Nikolas Luke 2018-09-07 03:10:04 MDT

Created attachment 7783 [details]
slurm.epilog Script

Comment 15 Nikolas Luke 2018-09-07 03:11:51 MDT

The problem has not recurred lately. Is the output of sdiag interesting from the nodes or the controller?

cat /etc/sysconfig/selinux
SELINUX=disabled
SELINUXTYPE=targeted

ss -p output of its-cs107 is attached.

We use the standard epilog.script provided by slurm, but i attached it, too.

I will definitely be back when it appears again.

Comment 16 Jason Booth 2018-09-12 11:48:46 MDT

Hi Nikolas,

> Is the output of sdiag interesting from the nodes or the controller?

The command 'sdiag' can run anywhere. It will report the same information about the scheduler regardless if it is ran on the node or on the controller.

The information I would also like to see would be debug logs from the slurmctld and the slurmd once you see the problem again, Comment #12.

-Jason

Comment 17 Jason Booth 2018-09-26 09:09:12 MDT

Hi Nikolas,

 Were you able to gather any additional information such as the output of sdiag during an event?

-Jason

Comment 18 Nikolas Luke 2018-10-04 01:23:04 MDT

The problem has not occurred again. I would say we close the ticket and I will contact you when I have the requested information.

Comment 19 Jason Booth 2018-10-04 09:22:03 MDT

> The problem has not occurred again. I would say we close the ticket and I will contact you when I have the requested information.

I will go ahead and close this as cannot reproduce.