Ticket 14134

Summary:	salloc --x11 followed by executing, say, xeyes, fails if job is submitted from the control node
Product:	Slurm	Reporter:	selva.nair
Component:	slurmstepd	Assignee:	Jacob Jenson <jacob>
Status:	RESOLVED INVALID	QA Contact:
Severity:	6 - No support contract
Priority:	---
Version:	21.08.5
Hardware:	Linux
OS:	Linux
Site:	-Other-	Slinky Site:	---
Alineos Sites:	---	Atos/Eviden Sites:	---
Confidential Site:	---	Coreweave sites:	---
Cray Sites:	---	DS9 clusters:	---
Google sites:	---	HPCnow Sites:	---
HPE Sites:	---	IBM Sites:	---
NOAA SIte:	---	NoveTech Sites:	---
Nvidia HWinf-CS Sites:	---	OCF Sites:	---
Recursion Pharma Sites:	---	SFW Sites:	---
SNIC sites:	---	Tzag Elita Sites:	---
Linux Distro:	---	Machine Name:
CLE Version:		Version Fixed:
Target Release:	---	DevPrio:	---
Emory-Cloud Sites:	---

Description selva.nair 2022-05-21 23:20:54 MDT

Note: though this is suspiciously similar to 13342, I'm not sure of the rules here, so posting as a new bug -- please merge with it if desired.

slurmd.conf has
LaunchParameters=use_interactive_step
PrologFlags=X11
InteractiveStepOptions="--interactive --pty --preserve-env --mpi=none $SHELL"

"salloc --x11" runs successfully and connects the user to the allocated node. But running any X program from the node fails with 
"Error: Can't open display: localhost:xx.0". 
DISPLAY gets correctly set, xauth has the right cookie copied from source host, so nothing looks obviously wrong. slurm logs show this error:

[2022-05-22T00:42:29.439] [91.extern] error: _x11_socket_read: slurm_open_msg_conn: Connection refused

Interestingly, everything works fine if the job is submitted from a host other than the slurm control host. This was tested by ensuring the job lands in the same compute node. 

Mine is a smallish cluster and the login node is also the slurm control node.

Before anyone asks: Not using --x11, but running  "salloc /usr/bin/bash" followed by "ssh -X $SLURM_NODELIST xeyes" does work irrespective of the submission node.

Comment 1 selva.nair 2022-05-22 09:06:12 MDT

Solved: all nodes had a line in /etc/hosts as

127.0.1.1  <nodename>

Ubuntu adds this when the hostname is set. Removing this on the controller fixes the issue. 

I guess this entry makes the controller provide 127.0.1.1 as the IP of the starting host to slurmstepd. Not sure why only x11 forwarding is affected.