Ticket 3927

Summary: Backup Controller does not start jobs because agent thread is not running
Product: Slurm Reporter: Thomas Opfer <hrz>
Component: ContributionsAssignee: Jacob Jenson <jacob>
Status: RESOLVED DUPLICATE QA Contact:
Severity: 6 - No support contract    
Priority: --- CC: hrz
Version: 17.02.5   
Hardware: Linux   
OS: Linux   
Site: -Other- Slinky Site: ---
Alineos Sites: --- Atos/Eviden Sites: ---
Confidential Site: --- Coreweave sites: ---
Cray Sites: --- DS9 clusters: ---
Google sites: --- HPCnow Sites: ---
HPE Sites: --- IBM Sites: ---
NOAA SIte: --- NoveTech Sites: ---
Nvidia HWinf-CS Sites: --- OCF Sites: ---
Recursion Pharma Sites: --- SFW Sites: ---
SNIC sites: --- Tzag Elita Sites: ---
Linux Distro: --- Machine Name: Lichtenberg High Performance Computer
CLE Version: Version Fixed:
Target Release: --- DevPrio: ---
Emory-Cloud Sites: ---
Attachments: Make the backup controller start jobs correctly.

Description Thomas Opfer 2017-06-25 08:11:26 MDT 
Created attachment 4829 [details]
Make the backup controller start jobs correctly.

We saw that jobs dispatched from the backup controller had state "Prolog". After some investigations, I realized that an agent thread disappeared.

I created a preliminary fix and attached it. I don't know whether it has other side effects, but I don't think so.
Comment 2 Danny Auble 2017-06-28 11:44:39 MDT 
Thomas, could you send us how to reproduce this?  I am not seeing a way the agent thread could disappear.
Comment 3 Thomas Opfer 2017-06-28 14:36:51 MDT 
(In reply to Danny Auble from comment #2)
> Thomas, could you send us how to reproduce this?  I am not seeing a way the
> agent thread could disappear.

This happens on our live system and on our test system when we call "scontrol takeover". No job dispatched from the backup controller starts correctly.

According to Totalview, on the debug controller it runs into this code and terminates the thread:

		if (slurmctld_config.shutdown_time) {
			slurm_mutex_unlock(&pending_mutex);
			break;
		}
Comment 4 Danny Auble 2017-06-29 09:56:26 MDT 
Thomas, I am unable to reproduce this.  Things failover just as you would expect.