Ticket 12565

Summary:	preempt events no longer logged by slurmctld
Product:	Slurm	Reporter:	Shawn Hoopes <shawn>
Component:	slurmctld	Assignee:	Scott Hilton <scott>
Status:	RESOLVED INFOGIVEN	QA Contact:
Severity:	4 - Minor Issue
Priority:	---	CC:	scott
Version:	21.08.1
Hardware:	Linux
OS:	Linux
Site:	SchedMD	Alineos Sites:	---
Atos/Eviden Sites:	---	Confidential Site:	---
Coreweave sites:	---	Cray Sites:	---
DS9 clusters:	---	HPCnow Sites:	---
HPE Sites:	---	IBM Sites:	---
NOAA SIte:	---	OCF Sites:	---
Recursion Pharma Sites:	---	SFW Sites:	---
SNIC sites:	---	Linux Distro:	---
Machine Name:		CLE Version:
Version Fixed:		Target Release:	---
DevPrio:	---	Emory-Cloud Sites:	---

Description Shawn Hoopes 2021-09-28 12:20:40 MDT

Comment 1 Shawn Hoopes 2021-09-28 12:22:33 MDT

With Slurm v20.8.x I used to be able to see preemption events in the slurmctld log file:

Nov 27 02:23:59 ip-172-31-41-226 043c52764139[8212]: slurmctld: preempted JobId=3 has been requeued to reclaim resources for JobId=4

In Slurm v21.08.1, at the same loglevel (debug3) I no longer see the events in the log

Thanks

Comment 2 Scott Hilton 2021-09-28 13:09:03 MDT

Shawn, 

It still works for me with this. What is your preempt setup?
PreemptMode=REQUEUE
PreemptType=preempt/partition_prio

-Scott

Comment 3 Shawn Hoopes 2021-09-28 13:35:36 MDT

Sorry Scott

Should have said this is QOS based preemption

Comment 4 Scott Hilton 2021-09-28 14:16:57 MDT

Shawn,

I'm still getting this in my slurmctld.log:
>[2021-09-28T14:11:57.543] preempted JobId=6202 has been requeued to reclaim resources for JobId=6203

With these settings:
>PreemptMode=Requeue
>PreemptType=preempt/qos
> 
>$sacctmgr list qos format=name,preemptMode,preempt
>      Name PreemptMode    Preempt 
>---------- ----------- ---------- 
>    normal     requeue                      
>     myqos     requeue     normal 

-Scott

Comment 6 Scott Hilton 2021-09-28 15:45:20 MDT

Closing ticket. This was not an issue with slurm.