Ticket 3927 - Backup Controller does not start jobs because agent thread is not running
Summary: Backup Controller does not start jobs because agent thread is not running
Status: RESOLVED DUPLICATE of ticket 5511
Alias: None
Product: Slurm
Classification: Unclassified
Component: Contributions (show other tickets)
Version: 17.02.5
Hardware: Linux Linux
: 6 - No support contract
Assignee: Jacob Jenson
QA Contact:
URL:
Depends on:
Blocks:
 
Reported: 2017-06-25 08:11 MDT by Thomas Opfer
Modified: 2018-08-16 16:37 MDT (History)
1 user (show)

See Also:
Site: -Other-
Slinky Site: ---
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
Google sites: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
NoveTech Sites: ---
Nvidia HWinf-CS Sites: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Tzag Elita Sites: ---
Linux Distro: ---
Machine Name: Lichtenberg High Performance Computer
CLE Version:
Version Fixed:
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---

Attachments
Make the backup controller start jobs correctly. (629 bytes, patch)
2017-06-25 08:11 MDT, Thomas Opfer 		Details | Diff
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this ticket.
Description Thomas Opfer 2017-06-25 08:11:26 MDT 
Created attachment 4829 [details]
Make the backup controller start jobs correctly.

We saw that jobs dispatched from the backup controller had state "Prolog". After some investigations, I realized that an agent thread disappeared.

I created a preliminary fix and attached it. I don't know whether it has other side effects, but I don't think so.
Comment 2 Danny Auble 2017-06-28 11:44:39 MDT 
Thomas, could you send us how to reproduce this?  I am not seeing a way the agent thread could disappear.
Comment 3 Thomas Opfer 2017-06-28 14:36:51 MDT 
(In reply to Danny Auble from comment #2)
> Thomas, could you send us how to reproduce this?  I am not seeing a way the
> agent thread could disappear.

This happens on our live system and on our test system when we call "scontrol takeover". No job dispatched from the backup controller starts correctly.

According to Totalview, on the debug controller it runs into this code and terminates the thread:

		if (slurmctld_config.shutdown_time) {
			slurm_mutex_unlock(&pending_mutex);
			break;
		}
Comment 4 Danny Auble 2017-06-29 09:56:26 MDT 
Thomas, I am unable to reproduce this.  Things failover just as you would expect.