Ticket 12565 - preempt events no longer logged by slurmctld
Summary: preempt events no longer logged by slurmctld
Status: RESOLVED INFOGIVEN
Alias: None
Product: Slurm
Classification: Unclassified
Component: slurmctld (show other tickets)
Version: 21.08.1
Hardware: Linux Linux
: 4 - Minor Issue
Assignee: Scott Hilton
QA Contact:
URL:
Depends on:
Blocks:
 
Reported: 2021-09-28 12:20 MDT by Shawn Hoopes
Modified: 2021-09-28 15:45 MDT (History)
1 user (show)

See Also:
Site: SchedMD
Slinky Site: ---
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
Google sites: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
NoveTech Sites: ---
Nvidia HWinf-CS Sites: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Tzag Elita Sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed:
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this ticket.
Description Shawn Hoopes 2021-09-28 12:20:40 MDT 

    


    


      



        
          Comment 1
        

        
          Shawn Hoopes

        

        
        

        
          2021-09-28 12:22:33 MDT
        

      






With Slurm v20.8.x I used to be able to see preemption events in the slurmctld log file:

Nov 27 02:23:59 ip-172-31-41-226 043c52764139[8212]: slurmctld: preempted JobId=3 has been requeued to reclaim resources for JobId=4

In Slurm v21.08.1, at the same loglevel (debug3) I no longer see the events in the log

Thanks

    


    


      



        
          Comment 2
        

        
          Scott Hilton

        

        
        

        
          2021-09-28 13:09:03 MDT
        

      






Shawn, 

It still works for me with this. What is your preempt setup?
PreemptMode=REQUEUE
PreemptType=preempt/partition_prio

-Scott

    


    


      



        
          Comment 3
        

        
          Shawn Hoopes

        

        
        

        
          2021-09-28 13:35:36 MDT
        

      






Sorry Scott

Should have said this is QOS based preemption

    


    


      



        
          Comment 4
        

        
          Scott Hilton

        

        
        

        
          2021-09-28 14:16:57 MDT
        

      






Shawn,

I'm still getting this in my slurmctld.log:
>[2021-09-28T14:11:57.543] preempted JobId=6202 has been requeued to reclaim resources for JobId=6203

With these settings:
>PreemptMode=Requeue
>PreemptType=preempt/qos
> 
>$sacctmgr list qos format=name,preemptMode,preempt
>      Name PreemptMode    Preempt 
>---------- ----------- ---------- 
>    normal     requeue                      
>     myqos     requeue     normal 

-Scott

    


    


      



        
          Comment 6
        

        
          Scott Hilton

        

        
        

        
          2021-09-28 15:45:20 MDT
        

      






Closing ticket. This was not an issue with slurm.