I suspect I'm just missing a config thing here, but preempt/qos doesn't seem to be working for us right now. We have: [day36@opal186:~]$ scontrol show config | grep -i preempt PreemptMode = CANCEL PreemptType = preempt/qos [day36@opal186:~]$ sacctmgr show qos Name Priority GraceTime Preempt PreemptMode Flags UsageThres UsageFactor GrpTRES GrpTRESMins GrpTRESRunMin GrpJobs GrpSubmit GrpWall MaxTRES MaxTRESPerNode MaxTRESMins MaxWall MaxTRESPU MaxJobsPU MaxSubmitPU MaxTRESPA MaxJobsPA MaxSubmitPA MinTRES ---------- ---------- ---------- ---------- ----------- ---------------------------------------- ---------- ----------- ------------- ------------- ------------- ------- --------- ----------- ------------- -------------- ------------- ----------- ------------- --------- ----------- ------------- --------- ----------- ------------- normal 1000000 00:00:00 standby cluster 1.000000 standby 1 00:00:00 cluster NoReserve,PartitionMaxNodes,OverPartQOS+ 1.000000 ... but when I try to preempt a 'standby' job with a 'normal' one, the 'normal' job just sits waiting on resources: [day36@opal186:~]$ squeue -p pdebug -o "%.18i %.9P %.8j %.8u %.2t %.10M %.6D %R %q" JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) QOS 43736 pdebug hostname day36 PD 0:00 32 (Resources) normal 43734 pdebug sleep day36 R 3:49 32 opal[1-32] standby Here's the slurmctld.log showing the 'normal' jobid 43736 waiting until 43734 exits normally after its sleep finishes: [2018-12-06T14:32:52.894] sched: _slurm_rpc_allocate_resources JobId=43736 NodeList=(null) usec=264 [2018-12-06T14:32:58.642] backfill: Failed to start JobId=43736 with opal[1-32] avail: Requested nodes are busy [2018-12-06T14:39:09.828] _job_complete: JobID=43734 State=0x1 NodeCnt=32 WEXITSTATUS 0 [2018-12-06T14:39:09.829] _job_complete: JobID=43734 State=0x8003 NodeCnt=32 done [2018-12-06T14:39:14.211] sched: Allocate JobID=43736 NodeList=opal[1-32] #CPUs=1152 Partition=pdebug [2018-12-06T14:39:14.693] _job_complete: JobID=43736 State=0x1 NodeCnt=32 WEXITSTATUS 0 [2018-12-06T14:39:14.694] _job_complete: JobID=43736 State=0x8003 NodeCnt=32 done
Hi Ryan, Can you check that the Preempt field is setup correctly in sacctmgr. jason@nh-blue:~/slurm/master$ sacctmgr list qos format=name,priority,preempt Name Priority Preempt ---------- ---------- ---------- normal 0 small 100000 big big 0 Preempt Other QOS' this QOS can preempt. You may be missing this step: e.g. sacctmgr modify qos small set preempt=big -Jason
Hi Jason, The 'normal' qos does have preempt over 'standby'. Preemption was working for us under 17.02, but doesn't seem to be working any more. [day36@opal186:jobdatalua]$ sacctmgr list qos format=name,priority,preempt Name Priority Preempt ---------- ---------- ---------- normal 1000000 standby standby 1 expedite 2000000 standby exempt 1000000 standby pdebug_qu+ 1000000 [day36@opal186:jobdatalua]$ (In reply to Jason Booth from comment #1) > Hi Ryan, > > Can you check that the Preempt field is setup correctly in sacctmgr. > > jason@nh-blue:~/slurm/master$ sacctmgr list qos format=name,priority,preempt > Name Priority Preempt > ---------- ---------- ---------- > normal 0 > small 100000 big > big 0 > > > Preempt > Other QOS' this QOS can preempt. > > > You may be missing this step: > e.g. > sacctmgr modify qos small set preempt=big > > -Jason
Hi Ryan, I ran a few tests with the configuration you mentioned on 17.11.8 but could not recreate the issue. Preemption worked for me. Please attach your slurm.conf, and the output of sprio. This will help me verify that the preemptor is a higher priority and that I am not overlooking anything from your configuration. -Jason
Created attachment 8560 [details] slum.conf
Created attachment 8561 [details] file included in slurm.conf
I'm switching things slightly on you here. The testbed cluster that I used for my first example is running priority/basic, so sprio doesn't do anything. I originally saw this on a production cluster running priority/multifactor though, so I've attached the slurm.conf from that cluster. Here's what the sprio, etc output look like on that cluster: [day36@pascal83:~]$ squeue -p pvis -o "%i %R %q" JOBID NODELIST(REASON) QOS 190499 (Resources) exempt 190498 pascal[1,3-13,15-20,22-31] standby [day36@pascal83:~]$ squeue -p pvis -o "%i %R %q" JOBID NODELIST(REASON) QOS 190499 (Resources) exempt 190498 pascal[1,3-13,15-20,22-31] standby [day36@pascal83:~]$ sprio JOBID PARTITION PRIORITY AGE FAIRSHARE QOS 190499 pvis 1082689 1 82689 1000000 [day36@pascal83:~]$ sacctmgr list qos format=name,priority,preempt Name Priority Preempt ---------- ---------- ---------- normal 1000000 standby standby 1 expedite 2000000 standby exempt 1000000 standby pdebug_qu+ 1000000 pdebug_ca+ 1000000 [day36@pascal83:~]$ I'll note that the 'pvis' partition is not the default partition, but it's otherwise set up the same as the default partition. Could that cause problems?
Hi Ryan, I was able to duplicate this issue and it was fixed in the following commit: d103d20039f4dad6b8b196400213fb08c2cbe04a I would highly suggest that you upgrade to at least 17.11.12 since there are a number of issues that have been fixed since 17.11.8. -Jason
Closing this out for now since this is known issue that was fixed in 17.11.10 *** This ticket has been marked as a duplicate of ticket 5293 ***
