Ticket 12565

Summary: preempt events no longer logged by slurmctld
Product: Slurm Reporter: Shawn Hoopes <shawn>
Component: slurmctldAssignee: Scott Hilton <scott>
Status: RESOLVED INFOGIVEN QA Contact:
Severity: 4 - Minor Issue    
Priority: --- CC: scott
Version: 21.08.1   
Hardware: Linux   
OS: Linux   
Site: SchedMD Slinky Site: ---
Alineos Sites: --- Atos/Eviden Sites: ---
Confidential Site: --- Coreweave sites: ---
Cray Sites: --- DS9 clusters: ---
Google sites: --- HPCnow Sites: ---
HPE Sites: --- IBM Sites: ---
NOAA SIte: --- NoveTech Sites: ---
Nvidia HWinf-CS Sites: --- OCF Sites: ---
Recursion Pharma Sites: --- SFW Sites: ---
SNIC sites: --- Tzag Elita Sites: ---
Linux Distro: --- Machine Name:
CLE Version: Version Fixed:
Target Release: --- DevPrio: ---
Emory-Cloud Sites: ---

Description Shawn Hoopes 2021-09-28 12:20:40 MDT 

    


    


      



        
          Comment 1
        

        
          Shawn Hoopes

        

        
        

        
          2021-09-28 12:22:33 MDT
        

      






With Slurm v20.8.x I used to be able to see preemption events in the slurmctld log file:

Nov 27 02:23:59 ip-172-31-41-226 043c52764139[8212]: slurmctld: preempted JobId=3 has been requeued to reclaim resources for JobId=4

In Slurm v21.08.1, at the same loglevel (debug3) I no longer see the events in the log

Thanks

    


    


      



        
          Comment 2
        

        
          Scott Hilton

        

        
        

        
          2021-09-28 13:09:03 MDT
        

      






Shawn, 

It still works for me with this. What is your preempt setup?
PreemptMode=REQUEUE
PreemptType=preempt/partition_prio

-Scott

    


    


      



        
          Comment 3
        

        
          Shawn Hoopes

        

        
        

        
          2021-09-28 13:35:36 MDT
        

      






Sorry Scott

Should have said this is QOS based preemption

    


    


      



        
          Comment 4
        

        
          Scott Hilton

        

        
        

        
          2021-09-28 14:16:57 MDT
        

      






Shawn,

I'm still getting this in my slurmctld.log:
>[2021-09-28T14:11:57.543] preempted JobId=6202 has been requeued to reclaim resources for JobId=6203

With these settings:
>PreemptMode=Requeue
>PreemptType=preempt/qos
> 
>$sacctmgr list qos format=name,preemptMode,preempt
>      Name PreemptMode    Preempt 
>---------- ----------- ---------- 
>    normal     requeue                      
>     myqos     requeue     normal 

-Scott

    


    


      



        
          Comment 6
        

        
          Scott Hilton

        

        
        

        
          2021-09-28 15:45:20 MDT
        

      






Closing ticket. This was not an issue with slurm.