Ticket 1231

Summary:	Jobs vanished when submitted to non-existent partition
Product:	Slurm	Reporter:	Akmal Madzlan <akmalm>
Component:	slurmctld	Assignee:	Brian Christiansen <brian>
Status:	RESOLVED INFOGIVEN	QA Contact:
Severity:	2 - High Impact
Priority:	---	CC:	brian, da
Version:	14.03.9
Hardware:	Linux
OS:	Linux
Site:	DownUnder GeoSolutions	Alineos Sites:	---
Atos/Eviden Sites:	---	Confidential Site:	---
Coreweave sites:	---	Cray Sites:	---
DS9 clusters:	---	HPCnow Sites:	---
HPE Sites:	---	IBM Sites:	---
NOAA SIte:	---	OCF Sites:	---
Recursion Pharma Sites:	---	SFW Sites:	---
SNIC sites:	---	Linux Distro:	---
Machine Name:		CLE Version:
Version Fixed:		Target Release:	---
DevPrio:	---	Emory-Cloud Sites:	---

Description Akmal Madzlan 2014-11-04 13:10:02 MST

Jobs vanished when submitted to 3 partitions, 2 of which still existed. Some of the jobs went into SE and we're speculating that the jobs that are currently running when the partition disappear/removed is put into SE state and the one that is still pending is vanished.

I think slurm should put the pending jobs into the other partitions specified or put it in some sort of error state when one a partition specified is removed. That way we can fix the error and then bring them back into the queue

[2014-10-31T22:06:56.198] error: Invalid partition (teambond,idle,desktopBigMem) for job 2202158
[2014-10-31T22:06:56.220] error: Invalid partition (teambond,idle,desktopBigMem) for job 2202159
[2014-10-31T22:06:56.236] error: Invalid partition (teambond,idle,desktopBigMem) for job 2202160
[2014-10-31T22:06:56.253] error: Invalid partition (teambond,idle,desktopBigMem) for job 2202161
[2014-10-31T22:06:56.277] error: Invalid partition (teambond,idle,desktopBigMem) for job 2202162
[2014-10-31T22:06:56.292] error: Invalid partition (teambond,idle,desktopBigMem) for job 2202163
[2014-10-31T22:06:56.310] error: Invalid partition (teambond,idle,desktopBigMem) for job 2202164
[2014-10-31T22:06:56.328] error: Invalid partition (teambond,idle,desktopBigMem) for job 2202165
[2014-10-31T22:06:56.346] error: Invalid partition (teambond,idle,desktopBigMem) for job 2202166
[2014-10-31T22:06:56.366] error: Invalid partition (teambond,idle,desktopBigMem) for job 2202167
[2014-10-31T22:06:56.389] error: Invalid partition (teambond,idle,desktopBigMem) for job 2202168
[2014-10-31T22:06:56.408] error: Invalid partition (teambond,idle,desktopBigMem) for job 2202169
[2014-10-31T22:06:56.427] error: Invalid partition (teambond,idle,desktopBigMem) for job 2202170
[2014-10-31T22:06:56.443] error: Invalid partition (teambond,idle,desktopBigMem) for job 2202171
[2014-10-31T22:06:56.464] error: Invalid partition (teambond,idle,desktopBigMem) for job 2202172
[2014-10-31T22:06:56.486] error: Invalid partition (teambond,idle,desktopBigMem) for job 2202173
[2014-10-31T22:06:56.506] error: Invalid partition (teambond,idle,desktopBigMem) for job 2202174

Comment 1 Brian Christiansen 2014-11-05 07:42:07 MST

I've been able to reproduce this is if I remove a partition from the slurm.conf and reconfigure. If you use "scontrol delete PartitionName=<partiton>", there is logic to prevent the partition from being deleted if the partition is in use by any job.

I'm looking into a solution for not canceling the jobs on a restart/reconfigure.

Comment 2 Brian Christiansen 2014-11-06 07:36:33 MST

The solution to prevent the jobs from being canceled will be considered for 15.08 -- It's a large project. The solution for now is to use "scontrol delete partiton=<partition>" before removing a partition from the slurm.conf and restarting. This will help ensure that there are no jobs referencing the partition at removal.

The FAQ has been updated with this information as well.
https://github.com/SchedMD/slurm/commit/2f4a18fa466ee55baa205390f154e75269074b5f

Let us know if you have any questions.

Thanks,
Brian

Comment 3 Akmal Madzlan 2014-11-09 14:37:21 MST

Thanks Brian