| Summary: | preempt events no longer logged by slurmctld | ||
|---|---|---|---|
| Product: | Slurm | Reporter: | Shawn Hoopes <shawn> |
| Component: | slurmctld | Assignee: | Scott Hilton <scott> |
| Status: | RESOLVED INFOGIVEN | QA Contact: | |
| Severity: | 4 - Minor Issue | ||
| Priority: | --- | CC: | scott |
| Version: | 21.08.1 | ||
| Hardware: | Linux | ||
| OS: | Linux | ||
| Site: | SchedMD | Slinky Site: | --- |
| Alineos Sites: | --- | Atos/Eviden Sites: | --- |
| Confidential Site: | --- | Coreweave sites: | --- |
| Cray Sites: | --- | DS9 clusters: | --- |
| Google sites: | --- | HPCnow Sites: | --- |
| HPE Sites: | --- | IBM Sites: | --- |
| NOAA SIte: | --- | NoveTech Sites: | --- |
| Nvidia HWinf-CS Sites: | --- | OCF Sites: | --- |
| Recursion Pharma Sites: | --- | SFW Sites: | --- |
| SNIC sites: | --- | Tzag Elita Sites: | --- |
| Linux Distro: | --- | Machine Name: | |
| CLE Version: | Version Fixed: | ||
| Target Release: | --- | DevPrio: | --- |
| Emory-Cloud Sites: | --- | ||
|
Description
Shawn Hoopes
2021-09-28 12:20:40 MDT
With Slurm v20.8.x I used to be able to see preemption events in the slurmctld log file: Nov 27 02:23:59 ip-172-31-41-226 043c52764139[8212]: slurmctld: preempted JobId=3 has been requeued to reclaim resources for JobId=4 In Slurm v21.08.1, at the same loglevel (debug3) I no longer see the events in the log Thanks Shawn, It still works for me with this. What is your preempt setup? PreemptMode=REQUEUE PreemptType=preempt/partition_prio -Scott Sorry Scott Should have said this is QOS based preemption Shawn, I'm still getting this in my slurmctld.log: >[2021-09-28T14:11:57.543] preempted JobId=6202 has been requeued to reclaim resources for JobId=6203 With these settings: >PreemptMode=Requeue >PreemptType=preempt/qos > >$sacctmgr list qos format=name,preemptMode,preempt > Name PreemptMode Preempt >---------- ----------- ---------- > normal requeue > myqos requeue normal -Scott Closing ticket. This was not an issue with slurm. |