Created attachment 4514 [details]
Slurm Log
I tried setting:
SchedulerParameters=bf_window=14400,assoc_limit_stop,bf_job_part_count_reserve=1
The main scheduler logic doesn't seem to start the small jobs now, but the small jobs are still backfilled. The scheduler keeps preempting jobs to try to start 11580, but it can't start, so the nodes are backfilled again. I'll attach that log too.
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) PRIORITY
11580 hpc test-sle hyaktest PD 0:00 6 (QOSGrpCpuLimit) 277
11573 hpc test-sle sjf4 R 9:46 1 n2201 275
11574 hpc test-sle sjf4 R 9:16 1 n2007 275
11575 hpc test-sle sjf4 R 8:46 1 n2093 275
11576 hpc test-sle sjf4 R 8:15 1 n2094 275
11577 hpc test-sle sjf4 R 8:13 1 n2143 275
11578 hpc test-sle sjf4 R 8:11 1 n2150 275
11579 hpc test-sle sjf4 R 8:09 1 n2154 275
11582 hpc test-sle sjf4 R 4:05 1 n2049 275
11583 hpc test-sle sjf4 R 4:05 1 n2052 275
11584 hpc test-sle sjf4 R 4:05 1 n2067 275
Even when they're very long:
Job ID Username Nodes Cores Memory Time Remaining
-------- -------- ----- ----- ------ --------------
11573 sjf4 1 28 100G 23:49:36
11574 sjf4 1 28 100G 23:50:06
11575 sjf4 1 28 100G 23:50:36
11576 sjf4 1 28 100G 23:51:07
11577 sjf4 1 28 100G 23:51:09
11578 sjf4 1 28 100G 23:51:11
11579 sjf4 1 28 100G 23:51:13
11582 sjf4 1 28 100G 6-23:55:17
11583 sjf4 1 28 100G 6-23:55:17
11584 sjf4 1 28 100G 6-23:55:17
Created attachment 4516 [details]
Slurm Log with Revised Scheduler Params
Hi slurm 17.02.3 will contain set of patches related to sched/backfill/qos/assoc_limit_stop. I will make some tests to be sure that new logic will work in your case. Could you send me slurmctld.log with Backfill DebugFlags? Dominik I submitted jobs under my userid sjf4, then the large one under hyaktest. I then cancelled 11691-11695 and it backfilled the pending jobs. It also should have not started any new jobs after I'd submitted 11690 I'd think.
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) PRIORITY
11690 hpc test-sle hyaktest PD 0:00 6 (Resources) 275
11696 hpc test-sle sjf4 PD 0:00 1 (Resources) 273
11697 hpc test-sle sjf4 PD 0:00 1 (Resources) 273
11698 hpc test-sle sjf4 PD 0:00 1 (Resources) 273
11699 hpc test-sle sjf4 PD 0:00 1 (Resources) 273
11700 hpc test-sle sjf4 PD 0:00 1 (Resources) 273
11691 hpc test-sle sjf4 R 3:17 1 n2057 273
11692 hpc test-sle sjf4 R 3:17 1 n2066 273
11693 hpc test-sle sjf4 R 3:17 1 n2068 273
11694 hpc test-sle sjf4 R 3:17 1 n2070 273
11695 hpc test-sle sjf4 R 3:17 1 n2072 273
11689 hpc test-sle sjf4 R 5:48 1 n2206 273
11687 hpc test-sle sjf4 R 6:18 1 n2059 273
11688 hpc test-sle sjf4 R 6:18 1 n2060 273
11683 hpc test-sle sjf4 R 6:48 1 n2154 273
11684 hpc test-sle sjf4 R 6:48 1 n2012 273
11685 hpc test-sle sjf4 R 6:48 1 n2071 273
11686 hpc test-sle sjf4 R 6:48 1 n2094 273
JobId=11690 JobName=test-sleep
UserId=hyaktest(383794) GroupId=hyak-hpc(202025) MCS_label=N/A
Priority=275 Nice=0 Account=hpc QOS=normal
JobState=PENDING Reason=QOSGrpCpuLimit Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
RunTime=00:00:00 TimeLimit=05:00:00 TimeMin=N/A
SubmitTime=2017-05-05T13:19:22 EligibleTime=2017-05-05T13:19:22
StartTime=Unknown EndTime=Unknown Deadline=N/A
PreemptTime=None SuspendTime=None SecsPreSuspend=0
Partition=hpc AllocNode:Sid=mgmt4:53509
ReqNodeList=(null) ExcNodeList=(null)
NodeList=(null) SchedNodeList=n[2002,2102-2106]
NumNodes=6-6 NumCPUs=6 NumTasks=6 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
TRES=cpu=6,mem=614400,node=6
Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
MinCPUsNode=1 MinMemoryNode=100G MinTmpDiskNode=0
Features=(null) DelayBoot=00:00:00
Gres=(null) Reservation=(null)
OverSubscribe=NO Contiguous=0 Licenses=(null) Network=(null)
Command=/gscratch/home/hyaktest/test-sleep.sh
WorkDir=/home/hyaktest
StdErr=/home/hyaktest/slurm-11690.out
StdIn=/dev/null
StdOut=/home/hyaktest/slurm-11690.out
Power=
The real world set of jobs that set me on this investigation are in the bkrs partition. Here's a subset of that display.
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) PRIORITY
11103 bkrs magnetic mjs5 PD 0:00 32 (QOSGrpCpuLimit) 293
11033 bkrs piping wdetmold PD 0:00 1 (QOSGrpCpuLimit) 200
11034 bkrs piping wdetmold PD 0:00 1 (QOSGrpCpuLimit) 200
11035 bkrs piping wdetmold PD 0:00 1 (QOSGrpCpuLimit) 200
11036 bkrs piping wdetmold PD 0:00 1 (QOSGrpCpuLimit) 200
11037 bkrs piping wdetmold PD 0:00 1 (QOSGrpCpuLimit) 200
11038 bkrs piping wdetmold PD 0:00 1 (QOSGrpCpuLimit) 200
11039 bkrs piping wdetmold PD 0:00 1 (QOSGrpCpuLimit) 200
11040 bkrs piping wdetmold PD 0:00 1 (QOSGrpCpuLimit) 200
11041 bkrs piping wdetmold PD 0:00 1 (QOSGrpCpuLimit) 200
11042 bkrs piping wdetmold PD 0:00 1 (QOSGrpCpuLimit) 200
11043 bkrs piping wdetmold PD 0:00 1 (QOSGrpCpuLimit) 200
11044 bkrs piping wdetmold PD 0:00 1 (QOSGrpCpuLimit) 200
11045 bkrs piping wdetmold PD 0:00 1 (QOSGrpCpuLimit) 200
11046 bkrs piping wdetmold PD 0:00 1 (QOSGrpCpuLimit) 200
11047 bkrs piping wdetmold PD 0:00 1 (QOSGrpCpuLimit) 200
11048 bkrs piping wdetmold PD 0:00 1 (QOSGrpCpuLimit) 200
11049 bkrs piping wdetmold PD 0:00 1 (QOSGrpCpuLimit) 200
11050 bkrs piping wdetmold PD 0:00 1 (QOSGrpCpuLimit) 200
11032 bkrs piping wdetmold R 15:41 1 n2119 200
11031 bkrs piping wdetmold R 43:13 1 n2142 200
11030 bkrs piping wdetmold R 1:07:26 1 n2149 200
11029 bkrs piping wdetmold R 1:08:57 1 n2164 200
11028 bkrs piping wdetmold R 1:22:12 1 n2108 200
11027 bkrs piping wdetmold R 1:30:42 1 n2179 200
11026 bkrs piping wdetmold R 1:38:43 1 n2178 200
11025 bkrs piping wdetmold R 2:09:43 1 n2005 200
11024 bkrs piping wdetmold R 3:08:55 1 n2080 200
11023 bkrs piping wdetmold R 3:11:55 1 n2161 200
11022 bkrs piping wdetmold R 3:16:13 1 n2054 200
11021 bkrs piping wdetmold R 3:29:43 1 n2077 200
11019 bkrs piping wdetmold R 3:38:56 1 n2159 200
JobId=11103 JobName=magnetic
UserId=mjs5(261746) GroupId=hyak-bkrs(20723) MCS_label=N/A
Priority=293 Nice=0 Account=bkrs QOS=normal
JobState=PENDING Reason=QOSGrpCpuLimit Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
RunTime=00:00:00 TimeLimit=1-00:00:00 TimeMin=N/A
SubmitTime=2017-05-01T20:37:08 EligibleTime=2017-05-01T20:37:08
StartTime=Unknown EndTime=Unknown Deadline=N/A
PreemptTime=None SuspendTime=None SecsPreSuspend=0
Partition=bkrs AllocNode:Sid=mox2:37967
ReqNodeList=(null) ExcNodeList=(null)
NodeList=(null) SchedNodeList=n[2012-2013,2019-2021,2033-2035,2062-2064,2070-2072,2083-2085,2102-2106,2114-2116,2130-2133,2180-2182]
NumNodes=32-32 NumCPUs=512 NumTasks=512 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
TRES=cpu=512,mem=3932160,node=32
Socks/Node=* NtasksPerN:B:S:C=16:0:*:* CoreSpec=*
MinCPUsNode=16 MinMemoryNode=120G MinTmpDiskNode=0
Features=(null) DelayBoot=00:00:00
Gres=(null) Reservation=(null)
OverSubscribe=NO Contiguous=0 Licenses=(null) Network=(null)
Command=/gscratch/bkrs/Magnetic/cl21_48_96_b6p3_m0p2416_m0p2050_Magnetic/slurm_qqq_mox_Magnetic_160MeV_b6p3_48_96.slurm
WorkDir=/gscratch/bkrs/Magnetic/cl21_48_96_b6p3_m0p2416_m0p2050_Magnetic
StdErr=/gscratch/bkrs/Magnetic/cl21_48_96_b6p3_m0p2416_m0p2050_Magnetic/jobfiles/slurm.%N.11103.err
StdIn=/dev/null
StdOut=/gscratch/bkrs/Magnetic/cl21_48_96_b6p3_m0p2416_m0p2050_Magnetic/jobfiles/slurm.%N.11103.out
Power=
I will attach the log with the backfill debugging momentarily.
Created attachment 4521 [details]
Slurmctld log with backfill debugging
Hi 17.02.3 should solve all of this issues. Dominik I updated to 17.02.3, but the behavior appears to be exactly the same. This is a serious bug for my site. Large jobs effectively will never start if there are enough small jobs competing for resources in the same floating partition. We have service level agreements with our customers that the right jobs will start and they'll start within a certain period of time, so we're not presently meeting those requirements. Can you provide a time frame as to when you can replicate this bug? If you can't, please let me know that so I can provide whatever information is necessary for you to do so. How often should I expect updates on your progress? When should I expect a patch to fix this issue? If this fix is not on your short list, I'm going to have to make other provisions like coming up with an alternate scheduler configuration. This is the one that was recommended to me by the SchedMD staff at SC15 however (bug 3013). I will attach a new log with backfill debugging and my current configuration (approximately the same as the old one). Test Procedure: I start 7 one node jobs (14777-14833) under my user id with a 1 day WC limit. I start 1 six node job (14485) under hyaktest with a 5 hour WC limit. I increase the priority of the job started as hyaktest: scontrol update JobId=14485 nice=-1000 I submit 10 additional jobs (14486-14495) under my userid with a 3 day WC limit. The additional jobs start via backfill even though this will prevent hyaktest's job from starting: JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) PRIORITY 14485 hpc test-sle hyaktest PD 0:00 6 (QOSGrpCpuLimit) 1279 14491 hpc test-sle sjf4 PD 0:00 1 (Priority) 278 14492 hpc test-sle sjf4 PD 0:00 1 (Priority) 278 14493 hpc test-sle sjf4 PD 0:00 1 (Priority) 278 14494 hpc test-sle sjf4 PD 0:00 1 (Priority) 278 14495 hpc test-sle sjf4 PD 0:00 1 (Priority) 278 14486 hpc test-sle sjf4 R 0:55 1 n2172 278 14487 hpc test-sle sjf4 R 0:55 1 n2043 278 14488 hpc test-sle sjf4 R 0:55 1 n2048 278 14489 hpc test-sle sjf4 R 0:55 1 n2053 278 14490 hpc test-sle sjf4 R 0:55 1 n2056 278 14479 hpc test-sle sjf4 R 11:54 1 n2149 278 14480 hpc test-sle sjf4 R 11:54 1 n2150 278 14481 hpc test-sle sjf4 R 11:54 1 n2001 278 14482 hpc test-sle sjf4 R 11:54 1 n2003 278 14483 hpc test-sle sjf4 R 11:54 1 n2009 278 14477 hpc test-sle sjf4 R 11:57 1 n2173 278 14478 hpc test-sle sjf4 R 11:57 1 n2148 278 I cancel 4 jobs (14477-14480) running under my userid with a 1 day WC limit. The additional jobs start even though again this will prevent the hyaktest job from starting: JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) PRIORITY 14485 hpc test-sle hyaktest PD 0:00 6 (QOSGrpCpuLimit) 1279 14495 hpc test-sle sjf4 PD 0:00 1 (Priority) 278 14491 hpc test-sle sjf4 R 0:17 1 n2173 278 14492 hpc test-sle sjf4 R 0:17 1 n2024 278 14493 hpc test-sle sjf4 R 0:17 1 n2026 278 14494 hpc test-sle sjf4 R 0:17 1 n2028 278 14486 hpc test-sle sjf4 R 2:48 1 n2172 278 14487 hpc test-sle sjf4 R 2:48 1 n2043 278 14488 hpc test-sle sjf4 R 2:48 1 n2048 278 14489 hpc test-sle sjf4 R 2:48 1 n2053 278 14490 hpc test-sle sjf4 R 2:48 1 n2056 278 14481 hpc test-sle sjf4 R 13:47 1 n2001 278 14482 hpc test-sle sjf4 R 13:47 1 n2003 278 14483 hpc test-sle sjf4 R 13:47 1 n2009 278 Created attachment 4587 [details]
Slurmctld 17023 log with backfill debugging
Created attachment 4588 [details]
Slurm 17023 configuration
Hi I can reproduce similar behavior. I can't give you time frames but I will try to attach some preliminary patch to the end of the day. Dominik Thank you. I appreciate knowing you've replicated the issue and that you're working on a new patch. Having something to pass back to customers experiencing this issue is important. Created attachment 4599 [details]
preliminary patch
Hi
This patch should help in this problem.
But it has some limitations, it works only for partition qos group limits.
General solution is more complicated and we need more time to prepare this.
Dominik
Ok. I understand. Thanks for the quick patch. I'll put this in early next week and let you know how it goes. The patch does not help the test case I've outlined. I see there's a new message for backfill debugging in the patch which indicates the code has done something, but we're not seeing that message. I'm not sure what the difference could be between our test setups, but I feel like I've outlined everything relevant here. I double checked to make sure I applied the patch correctly and I have. I'll attach the log from my latest test (jobs running in the hpc partition are the relevant ones). Created attachment 4612 [details]
Slurmctld backfill debugging log after initial patch
Dominik Could you send me: "scontrol show job" output? Dominik **** State after submitting single node jobs (12 hrs WC), submitting multi-node job (5 hrs WC), and adjusting priority on multi-node job ****
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) PRIORITY
15352 hpc test-sle hyaktest PD 0:00 6 (QOSGrpCpuLimit) 679
15351 hpc test-sle sjf4 R 2:37 1 n2145 278
15346 hpc test-sle sjf4 R 2:40 1 n2157 278
15347 hpc test-sle sjf4 R 2:40 1 n2141 278
15348 hpc test-sle sjf4 R 2:40 1 n2142 278
15349 hpc test-sle sjf4 R 2:40 1 n2143 278
15350 hpc test-sle sjf4 R 2:40 1 n2144 278
15345 hpc test-sle sjf4 R 2:43 1 n2138 278
JobId=15352 JobName=test-sleep
UserId=hyaktest(383794) GroupId=hyak-hpc(202025) MCS_label=N/A
Priority=679 Nice=-400 Account=hpc QOS=normal
JobState=PENDING Reason=QOSGrpCpuLimit Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
RunTime=00:00:00 TimeLimit=05:00:00 TimeMin=N/A
SubmitTime=2017-05-22T16:39:53 EligibleTime=2017-05-22T16:39:53
StartTime=2017-05-22T16:48:41 EndTime=2017-05-22T21:48:41 Deadline=N/A
PreemptTime=None SuspendTime=None SecsPreSuspend=0
Partition=hpc AllocNode:Sid=mgmt4:18604
ReqNodeList=(null) ExcNodeList=(null)
NodeList=(null) SchedNodeList=n[2116-2121]
NumNodes=6-6 NumCPUs=6 NumTasks=6 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
TRES=cpu=6,mem=614400,node=6
Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
MinCPUsNode=1 MinMemoryNode=100G MinTmpDiskNode=0
Features=(null) DelayBoot=00:00:00
Gres=(null) Reservation=(null)
OverSubscribe=NO Contiguous=0 Licenses=(null) Network=(null)
Command=/gscratch/home/hyaktest/test-sleep.sh
WorkDir=/home/hyaktest
StdErr=/home/hyaktest/slurm-15352.out
StdIn=/dev/null
StdOut=/home/hyaktest/slurm-15352.out
Power=
**** State after submitting additional single node jobs (48 hrs WC) and some are backfilled ****
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) PRIORITY
15352 hpc test-sle hyaktest PD 0:00 6 (QOSGrpCpuLimit) 679
15359 hpc test-sle sjf4 PD 0:00 1 (QOSGrpCpuLimit) 278
15360 hpc test-sle sjf4 PD 0:00 1 (QOSGrpCpuLimit) 278
15361 hpc test-sle sjf4 PD 0:00 1 (QOSGrpCpuLimit) 278
15362 hpc test-sle sjf4 PD 0:00 1 (QOSGrpCpuLimit) 278
15354 hpc test-sle sjf4 R 0:04 1 n2146 278
15355 hpc test-sle sjf4 R 0:04 1 n2147 278
15356 hpc test-sle sjf4 R 0:04 1 n2167 278
15357 hpc test-sle sjf4 R 0:04 1 n2168 278
15358 hpc test-sle sjf4 R 0:04 1 n2169 278
15351 hpc test-sle sjf4 R 5:07 1 n2145 278
15346 hpc test-sle sjf4 R 5:10 1 n2157 278
15347 hpc test-sle sjf4 R 5:10 1 n2141 278
15348 hpc test-sle sjf4 R 5:10 1 n2142 278
15349 hpc test-sle sjf4 R 5:10 1 n2143 278
15350 hpc test-sle sjf4 R 5:10 1 n2144 278
15345 hpc test-sle sjf4 R 5:13 1 n2138 278
JobId=15352 JobName=test-sleep
UserId=hyaktest(383794) GroupId=hyak-hpc(202025) MCS_label=N/A
Priority=679 Nice=-400 Account=hpc QOS=normal
JobState=PENDING Reason=QOSGrpCpuLimit Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
RunTime=00:00:00 TimeLimit=05:00:00 TimeMin=N/A
SubmitTime=2017-05-22T16:39:53 EligibleTime=2017-05-22T16:39:53
StartTime=2017-05-22T16:52:46 EndTime=2017-05-22T21:52:46 Deadline=N/A
PreemptTime=None SuspendTime=None SecsPreSuspend=0
Partition=hpc AllocNode:Sid=mgmt4:18604
ReqNodeList=(null) ExcNodeList=(null)
NodeList=(null) SchedNodeList=n[2116-2121]
NumNodes=6-6 NumCPUs=6 NumTasks=6 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
TRES=cpu=6,mem=614400,node=6
Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
MinCPUsNode=1 MinMemoryNode=100G MinTmpDiskNode=0
Features=(null) DelayBoot=00:00:00
Gres=(null) Reservation=(null)
OverSubscribe=NO Contiguous=0 Licenses=(null) Network=(null)
Command=/gscratch/home/hyaktest/test-sleep.sh
WorkDir=/home/hyaktest
StdErr=/home/hyaktest/slurm-15352.out
StdIn=/dev/null
StdOut=/home/hyaktest/slurm-15352.out
Power=
Job ID Username Nodes Cores Memory Time Remaining
-------- -------- ----- ----- ------ --------------
15345 sjf4 1 28 100G 11:53:03
15346 sjf4 1 28 100G 11:53:06
15347 sjf4 1 28 100G 11:53:06
15348 sjf4 1 28 100G 11:53:06
15349 sjf4 1 28 100G 11:53:06
15350 sjf4 1 28 100G 11:53:06
15351 sjf4 1 28 100G 11:53:09
15354 sjf4 1 28 100G 1-23:58:12
15355 sjf4 1 28 100G 1-23:58:12
15356 sjf4 1 28 100G 1-23:58:12
15357 sjf4 1 28 100G 1-23:58:12
15358 sjf4 1 28 100G 1-23:58:12
**** Cancelled 2 hr WC single node jobs and the waiting ones started *****
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) PRIORITY
15352 hpc test-sle hyaktest PD 0:00 6 (QOSGrpCpuLimit) 679
15359 hpc test-sle sjf4 R 0:01 1 n2146 278
15360 hpc test-sle sjf4 R 0:01 1 n2147 278
15361 hpc test-sle sjf4 R 0:01 1 n2167 278
15362 hpc test-sle sjf4 R 0:01 1 n2168 278
15351 hpc test-sle sjf4 R 9:04 1 n2145 278
15346 hpc test-sle sjf4 R 9:07 1 n2157 278
15347 hpc test-sle sjf4 R 9:07 1 n2141 278
15348 hpc test-sle sjf4 R 9:07 1 n2142 278
15349 hpc test-sle sjf4 R 9:07 1 n2143 278
15350 hpc test-sle sjf4 R 9:07 1 n2144 278
15345 hpc test-sle sjf4 R 9:10 1 n2138 278
JobId=15352 JobName=test-sleep
UserId=hyaktest(383794) GroupId=hyak-hpc(202025) MCS_label=N/A
Priority=679 Nice=-400 Account=hpc QOS=normal
JobState=PENDING Reason=QOSGrpCpuLimit Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
RunTime=00:00:00 TimeLimit=05:00:00 TimeMin=N/A
SubmitTime=2017-05-22T16:39:53 EligibleTime=2017-05-22T16:39:53
StartTime=2017-05-22T16:57:16 EndTime=2017-05-22T21:57:16 Deadline=N/A
PreemptTime=None SuspendTime=None SecsPreSuspend=0
Partition=hpc AllocNode:Sid=mgmt4:18604
ReqNodeList=(null) ExcNodeList=(null)
NodeList=(null) SchedNodeList=n[2116-2121]
NumNodes=6-6 NumCPUs=6 NumTasks=6 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
TRES=cpu=6,mem=614400,node=6
Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
MinCPUsNode=1 MinMemoryNode=100G MinTmpDiskNode=0
Features=(null) DelayBoot=00:00:00
Gres=(null) Reservation=(null)
OverSubscribe=NO Contiguous=0 Licenses=(null) Network=(null)
Command=/gscratch/home/hyaktest/test-sleep.sh
WorkDir=/home/hyaktest
StdErr=/home/hyaktest/slurm-15352.out
StdIn=/dev/null
StdOut=/home/hyaktest/slurm-15352.out
Power=
Job ID Username Nodes Cores Memory Time Remaining
-------- -------- ----- ----- ------ --------------
15345 sjf4 1 28 100G 11:49:48
15346 sjf4 1 28 100G 11:49:51
15347 sjf4 1 28 100G 11:49:51
15348 sjf4 1 28 100G 11:49:51
15349 sjf4 1 28 100G 11:49:51
15350 sjf4 1 28 100G 11:49:51
15351 sjf4 1 28 100G 11:49:54
15359 sjf4 1 28 100G 1-23:58:57
15360 sjf4 1 28 100G 1-23:58:57
15361 sjf4 1 28 100G 1-23:58:57
15362 sjf4 1 28 100G 1-23:58:57
Created attachment 4613 [details]
Patched slurmctld showing log when scontrol output taken
After thinking about it, I figured out you probably wanted all of the output for every job. I've repeated that test and I'll attach that output.
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) PRIORITY
15373 hpc test-sle hyaktest PD 0:00 6 (QOSGrpCpuLimit) 679
15379 hpc test-sle sjf4 PD 0:00 1 (Priority) 278
15380 hpc test-sle sjf4 PD 0:00 1 (Priority) 278
15381 hpc test-sle sjf4 PD 0:00 1 (Priority) 278
15374 hpc test-sle sjf4 R 0:42 1 n2150 278
15375 hpc test-sle sjf4 R 0:42 1 n2146 278
15376 hpc test-sle sjf4 R 0:42 1 n2147 278
15377 hpc test-sle sjf4 R 0:42 1 n2167 278
15378 hpc test-sle sjf4 R 0:42 1 n2168 278
15368 hpc test-sle sjf4 R 2:02 1 n2141 278
15369 hpc test-sle sjf4 R 2:02 1 n2142 278
15370 hpc test-sle sjf4 R 2:02 1 n2143 278
15371 hpc test-sle sjf4 R 2:02 1 n2144 278
15372 hpc test-sle sjf4 R 2:02 1 n2145 278
15366 hpc test-sle sjf4 R 2:05 1 n2138 278
15367 hpc test-sle sjf4 R 2:05 1 n2157 278
Created attachment 4614 [details]
scontrol show job output
Created attachment 4615 [details]
slurmctld log corresponding to scontrol show job
Hi Do you still use SchedulerParameters assoc_limit_stop? Dominik Yes, the configuration attached to the bug is the one we use. On May 23, 2017 3:16 AM, <bugs@schedmd.com> wrote: *Comment # 32 <https://bugs.schedmd.com/show_bug.cgi?id=3776#c32> on bug 3776 <https://bugs.schedmd.com/show_bug.cgi?id=3776> from Dominik Bartkiewicz <bart@schedmd.com> * Hi Do you still use SchedulerParameters assoc_limit_stop? Dominik ------------------------------ You are receiving this mail because: - You reported the bug. Hi Sorry for inconvenience. I am surprised that this patch didn't help. I was sure that I have fully recreated this locally. We prepared new patch which is more completed solution but with similar logic. I have found some mistakes in current patch witch can cause it doesn't work properly. Could you try with new patch and with enable debug flag Backfill and setdebug 5 and attach a chunk of the logs? Turn on: scontrol setdebugflag +backfill scontrol setdebug 5 Turn off: scontrol setdebugflag -backfill scontrol setdebug 2 Dominik Sorry, I forget to mention that "patch proposal v3" can be only applied to current git source from branch "slurm-17.02". (git checkout slurm-17.02) to be sure: - partition hpc uses qos hpc. Could you send me output from "scontrol show asso" and "sacctmgr show qos"? Dominik Created attachment 4635 [details]
show asso output
Created attachment 4636 [details]
show qos output
I attached that output you requested.
I'll build the updates today and install them tomorrow morning.
When you say:
> to be sure:
> - partition hpc uses qos hpc.
This is what you mean?
PartitionName=hpc Nodes=n2[001-184,195-202,206-210] Default=NO AllowAccounts=hpc QOS=hpc Priority=1000 DefaultTime=1:00:00 OverSubscribe=EXCLUSIVE PreemptMode=OFF
That's how I have it set.
This patch looks like it's fixed the issue. I'll attach the section of the logs with that debugging information shortly. Is this fix expected to show up in 17.02.4? Thanks a lot.
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) PRIORITY
16842 hpc test-sle hyaktest PD 0:00 6 (QOSGrpCpuLimit) 680
16843 hpc test-sle sjf4 PD 0:00 1 (Priority) 278
16844 hpc test-sle sjf4 PD 0:00 1 (Priority) 278
16845 hpc test-sle sjf4 PD 0:00 1 (Priority) 278
16846 hpc test-sle sjf4 PD 0:00 1 (Priority) 278
16847 hpc test-sle sjf4 PD 0:00 1 (Priority) 278
16848 hpc test-sle sjf4 PD 0:00 1 (Priority) 278
16849 hpc test-sle sjf4 PD 0:00 1 (Priority) 278
16850 hpc test-sle sjf4 PD 0:00 1 (Priority) 278
16851 hpc test-sle sjf4 PD 0:00 1 (Priority) 278
16837 hpc test-sle sjf4 R 2:15 1 n2144 278
16838 hpc test-sle sjf4 R 2:15 1 n2145 278
16839 hpc test-sle sjf4 R 2:15 1 n2010 278
16840 hpc test-sle sjf4 R 2:15 1 n2024 278
16841 hpc test-sle sjf4 R 2:15 1 n2039 278
16835 hpc test-sle sjf4 R 2:17 1 n2172 278
16836 hpc test-sle sjf4 R 2:17 1 n2143 278
16834 hpc test-sle sjf4 R 2:20 1 n2171 278
Created attachment 4652 [details]
slurmctld log after patch v3
Hi I am glad to hear that patch helped. It will be in 17.02.4. I will let you know when we add it to repo. Dominik Hi Now this fix with additional patches is in repo and will be included in 17.02.4. Thanks for patience Dominik |
Created attachment 4513 [details] Slurm Configuration We have a partition for each account/QOS with all the nodes of a given configuration. We then restrict the number of CPUs that account/QOS can use with a GrpTRES on the QOS: Name|GrpTRES| hpc|cpu=336,mem=2976G| Our floating partition set up starts small lower priority jobs at the expense of large higher priority jobs however. Job 11409 will not start if I cancel 5 of the single node jobs, then another 5. It will just keep starting the single node jobs owned by my userid and the 6 node job owned by hyaktest will never start as long as there's a stream of single node jobs. It doesn't look like the QOS limit and job priority are both being considered when starting jobs. Any suggestions you have are appreciated. JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) PRIORITY 11409 hpc test-sle hyaktest PD 0:00 6 (QOSGrpCpuLimit) 277 11401 hpc test-sle sjf4 R 11:25 1 n2017 276 11402 hpc test-sle sjf4 R 11:25 1 n2018 276 11403 hpc test-sle sjf4 R 11:25 1 n2019 276 11404 hpc test-sle sjf4 R 11:25 1 n2020 276 11405 hpc test-sle sjf4 R 11:25 1 n2021 276 11406 hpc test-sle sjf4 R 11:25 1 n2091 276 11410 hpc test-sle sjf4 R 10:46 1 n2139 276 11411 hpc test-sle sjf4 R 10:46 1 n2161 276 11412 hpc test-sle sjf4 R 10:46 1 n2162 276 11413 hpc test-sle sjf4 R 10:46 1 n2095 276 11414 hpc test-sle sjf4 R 10:46 1 n2176 276 11415 hpc test-sle sjf4 R 8:02 1 n2165 276 11416 hpc test-sle sjf4 PD 0:00 1 (QOSGrpCpuLimit) 276 11417 hpc test-sle sjf4 PD 0:00 1 (QOSGrpCpuLimit) 276 11418 hpc test-sle sjf4 PD 0:00 1 (QOSGrpCpuLimit) 276 11419 hpc test-sle sjf4 PD 0:00 1 (QOSGrpCpuLimit) 276 11420 hpc test-sle sjf4 PD 0:00 1 (QOSGrpCpuLimit) 276 11421 hpc test-sle sjf4 PD 0:00 1 (QOSGrpCpuLimit) 276 # scontrol show job 11409 JobId=11409 JobName=test-sleep UserId=hyaktest(383794) GroupId=hyak-hpc(202025) MCS_label=N/A Priority=277 Nice=0 Account=hpc QOS=normal JobState=PENDING Reason=QOSGrpCpuLimit Dependency=(null) Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:00:00 TimeLimit=1-00:00:00 TimeMin=N/A SubmitTime=2017-05-04T13:38:20 EligibleTime=2017-05-04T13:38:20 StartTime=Unknown EndTime=Unknown Deadline=N/A PreemptTime=None SuspendTime=None SecsPreSuspend=0 Partition=hpc AllocNode:Sid=mgmt4:11404 ReqNodeList=(null) ExcNodeList=(null) NodeList=(null) SchedNodeList=n[2002,2092-2096] NumNodes=6-6 NumCPUs=6 NumTasks=6 CPUs/Task=1 ReqB:S:C:T=0:0:*:* TRES=cpu=6,mem=614400,node=6 Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryNode=100G MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 Gres=(null) Reservation=(null) OverSubscribe=NO Contiguous=0 Licenses=(null) Network=(null) Command=/gscratch/home/hyaktest/test-sleep.sh WorkDir=/home/hyaktest StdErr=/home/hyaktest/slurm-11409.out StdIn=/dev/null StdOut=/home/hyaktest/slurm-11409.out Power=