Created attachment 7475 [details]
Proposed fix.

When running with slurmctld with a primary and backup controller it occasionally happens that after a failover the backup controller doesn't clean up finished jobs.
In the 'squeue -l' output, the finished jobs appear in 'COMPLETING' state while 'scontrol node <node>' shows the node the job was running on in state 'IDLE+COMPLETING'. This persists until the primary controller comes back in which case it takes care of the cleanup.
The backup controller needs to be restarted to fix this problem for future failover situations.

An analysis of this problem showed that this situation is caused by a race in the backup controller:
When the backup controller detects that the primary controller is inaccessible, it will run thru a restart cycle. To trigger the shutdown of some entities, it will set slurmctld_config.shutdown_time to a value != 0. Before continuing as the controller in charge, it resets this variable to 0 again.
The agent which handles the request queue - from a separate thread - wakes up periodically (in a 2 sec interval) and checks for things to do. If it finds slurmctld_config.shutdown_time set to a value != 0, it will  terminate.

If this wakeup occurs in the 'takeover window' between the variable being set to !=0 and reset to 0, the agent goes away and will no longer be available to handle queued requests as there is nothing at the end of the 'takeover window' that would restart it.

The proposed (See attachment) fix adds a restart of the agent by calling agent_init() afte r slurmctld_config.shutdown_time has been reset to 0. Should an agent still be running (because it didn't wake up during the 'takeover window') it will be caught in agent_init().

The same race problem does exist with 17.11.8, I have not been able to trigger the described problem, yet. An analysis showed that the 'race window' (ie the time during which slurmctld_config.shutdown_time is != 0) is a lot shorter.