Ticket 3776

Summary: Floating Partitions Start Small Jobs Constantly
Product: Slurm Reporter: Stephen Fralich <sjf4>
Component: SchedulingAssignee: Dominik Bartkiewicz <dbartkiewicz>
Status: RESOLVED FIXED QA Contact:
Severity: 2 - High Impact    
Priority: --- CC: dbartkiewicz
Version: 17.02.3   
Hardware: Linux   
OS: Linux   
Site: University of Washington Slinky Site: ---
Alineos Sites: --- Atos/Eviden Sites: ---
Confidential Site: --- Coreweave sites: ---
Cray Sites: --- DS9 clusters: ---
Google sites: --- HPCnow Sites: ---
HPE Sites: --- IBM Sites: ---
NOAA SIte: --- NoveTech Sites: ---
Nvidia HWinf-CS Sites: --- OCF Sites: ---
Recursion Pharma Sites: --- SFW Sites: ---
SNIC sites: --- Tzag Elita Sites: ---
Linux Distro: --- Machine Name:
CLE Version: Version Fixed: 17.02.4
Target Release: --- DevPrio: ---
Emory-Cloud Sites: ---
Attachments: Slurm Configuration
Slurm Log
Slurm Log with Revised Scheduler Params
Slurmctld log with backfill debugging
Slurmctld 17023 log with backfill debugging
Slurm 17023 configuration
preliminary patch
Slurmctld backfill debugging log after initial patch
Patched slurmctld showing log when scontrol output taken
scontrol show job output
slurmctld log corresponding to scontrol show job
patch proposal v3
show asso output
show qos output
slurmctld log after patch v3

Description Stephen Fralich 2017-05-04 15:58:30 MDT
Created attachment 4513 [details]
Slurm Configuration

We have a partition for each account/QOS with all the nodes of a given configuration. We then restrict the number of CPUs that account/QOS can use with a GrpTRES on the QOS:
Name|GrpTRES|
hpc|cpu=336,mem=2976G|

Our floating partition set up starts small lower priority jobs at the expense of large higher priority jobs however.

Job 11409 will not start if I cancel 5 of the single node jobs, then another 5. It will just keep starting the single node jobs owned by my userid and the 6 node job owned by hyaktest will never start as long as there's a stream of single node jobs.

It doesn't look like the QOS limit and job priority are both being considered when starting jobs. Any suggestions you have are appreciated.

             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON) PRIORITY
             11409       hpc test-sle hyaktest PD       0:00      6 (QOSGrpCpuLimit) 277
             11401       hpc test-sle     sjf4  R      11:25      1 n2017 276
             11402       hpc test-sle     sjf4  R      11:25      1 n2018 276
             11403       hpc test-sle     sjf4  R      11:25      1 n2019 276
             11404       hpc test-sle     sjf4  R      11:25      1 n2020 276
             11405       hpc test-sle     sjf4  R      11:25      1 n2021 276
             11406       hpc test-sle     sjf4  R      11:25      1 n2091 276
             11410       hpc test-sle     sjf4  R      10:46      1 n2139 276
             11411       hpc test-sle     sjf4  R      10:46      1 n2161 276
             11412       hpc test-sle     sjf4  R      10:46      1 n2162 276
             11413       hpc test-sle     sjf4  R      10:46      1 n2095 276
             11414       hpc test-sle     sjf4  R      10:46      1 n2176 276
             11415       hpc test-sle     sjf4  R       8:02      1 n2165 276
             11416       hpc test-sle     sjf4 PD       0:00      1 (QOSGrpCpuLimit) 276
             11417       hpc test-sle     sjf4 PD       0:00      1 (QOSGrpCpuLimit) 276
             11418       hpc test-sle     sjf4 PD       0:00      1 (QOSGrpCpuLimit) 276
             11419       hpc test-sle     sjf4 PD       0:00      1 (QOSGrpCpuLimit) 276
             11420       hpc test-sle     sjf4 PD       0:00      1 (QOSGrpCpuLimit) 276
             11421       hpc test-sle     sjf4 PD       0:00      1 (QOSGrpCpuLimit) 276

# scontrol show job 11409
JobId=11409 JobName=test-sleep
   UserId=hyaktest(383794) GroupId=hyak-hpc(202025) MCS_label=N/A
   Priority=277 Nice=0 Account=hpc QOS=normal
   JobState=PENDING Reason=QOSGrpCpuLimit Dependency=(null)
   Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
   RunTime=00:00:00 TimeLimit=1-00:00:00 TimeMin=N/A
   SubmitTime=2017-05-04T13:38:20 EligibleTime=2017-05-04T13:38:20
   StartTime=Unknown EndTime=Unknown Deadline=N/A
   PreemptTime=None SuspendTime=None SecsPreSuspend=0
   Partition=hpc AllocNode:Sid=mgmt4:11404
   ReqNodeList=(null) ExcNodeList=(null)
   NodeList=(null) SchedNodeList=n[2002,2092-2096]
   NumNodes=6-6 NumCPUs=6 NumTasks=6 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
   TRES=cpu=6,mem=614400,node=6
   Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
   MinCPUsNode=1 MinMemoryNode=100G MinTmpDiskNode=0
   Features=(null) DelayBoot=00:00:00
   Gres=(null) Reservation=(null)
   OverSubscribe=NO Contiguous=0 Licenses=(null) Network=(null)
   Command=/gscratch/home/hyaktest/test-sleep.sh
   WorkDir=/home/hyaktest
   StdErr=/home/hyaktest/slurm-11409.out
   StdIn=/dev/null
   StdOut=/home/hyaktest/slurm-11409.out
   Power=
Comment 1 Stephen Fralich 2017-05-04 16:05:54 MDT
Created attachment 4514 [details]
Slurm Log
Comment 2 Stephen Fralich 2017-05-04 17:56:27 MDT
I tried setting:
SchedulerParameters=bf_window=14400,assoc_limit_stop,bf_job_part_count_reserve=1

The main scheduler logic doesn't seem to start the small jobs now, but the small jobs are still backfilled. The scheduler keeps preempting jobs to try to start 11580, but it can't start, so the nodes are backfilled again. I'll attach that log too.

             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON) PRIORITY
             11580       hpc test-sle hyaktest PD       0:00      6 (QOSGrpCpuLimit) 277
             11573       hpc test-sle     sjf4  R       9:46      1 n2201 275
             11574       hpc test-sle     sjf4  R       9:16      1 n2007 275
             11575       hpc test-sle     sjf4  R       8:46      1 n2093 275
             11576       hpc test-sle     sjf4  R       8:15      1 n2094 275
             11577       hpc test-sle     sjf4  R       8:13      1 n2143 275
             11578       hpc test-sle     sjf4  R       8:11      1 n2150 275
             11579       hpc test-sle     sjf4  R       8:09      1 n2154 275
             11582       hpc test-sle     sjf4  R       4:05      1 n2049 275
             11583       hpc test-sle     sjf4  R       4:05      1 n2052 275
             11584       hpc test-sle     sjf4  R       4:05      1 n2067 275


Even when they're very long:
  Job ID    Username    Nodes    Cores    Memory    Time Remaining
--------    --------    -----    -----    ------    --------------
   11573        sjf4        1       28      100G          23:49:36
   11574        sjf4        1       28      100G          23:50:06
   11575        sjf4        1       28      100G          23:50:36
   11576        sjf4        1       28      100G          23:51:07
   11577        sjf4        1       28      100G          23:51:09
   11578        sjf4        1       28      100G          23:51:11
   11579        sjf4        1       28      100G          23:51:13
   11582        sjf4        1       28      100G        6-23:55:17
   11583        sjf4        1       28      100G        6-23:55:17
   11584        sjf4        1       28      100G        6-23:55:17
Comment 3 Stephen Fralich 2017-05-04 17:57:09 MDT
Created attachment 4516 [details]
Slurm Log with Revised Scheduler Params
Comment 4 Dominik Bartkiewicz 2017-05-05 07:16:39 MDT
Hi

slurm 17.02.3 will contain set of patches related to sched/backfill/qos/assoc_limit_stop. I will make some tests to be sure that new logic will work in your case.

Could you send me slurmctld.log with Backfill DebugFlags?

Dominik
Comment 5 Stephen Fralich 2017-05-05 14:37:12 MDT
I submitted jobs under my userid sjf4, then the large one under hyaktest. I then cancelled 11691-11695 and it backfilled the pending jobs. It also should have not started any new jobs after I'd submitted 11690 I'd think.

             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)             PRIORITY
             11690       hpc test-sle hyaktest PD       0:00      6 (Resources)                  275
             11696       hpc test-sle     sjf4 PD       0:00      1 (Resources)                  273
             11697       hpc test-sle     sjf4 PD       0:00      1 (Resources)                  273
             11698       hpc test-sle     sjf4 PD       0:00      1 (Resources)                  273
             11699       hpc test-sle     sjf4 PD       0:00      1 (Resources)                  273
             11700       hpc test-sle     sjf4 PD       0:00      1 (Resources)                  273
             11691       hpc test-sle     sjf4  R       3:17      1 n2057                  273
             11692       hpc test-sle     sjf4  R       3:17      1 n2066                  273
             11693       hpc test-sle     sjf4  R       3:17      1 n2068                  273
             11694       hpc test-sle     sjf4  R       3:17      1 n2070                  273
             11695       hpc test-sle     sjf4  R       3:17      1 n2072                  273
             11689       hpc test-sle     sjf4  R       5:48      1 n2206                  273
             11687       hpc test-sle     sjf4  R       6:18      1 n2059                  273
             11688       hpc test-sle     sjf4  R       6:18      1 n2060                  273
             11683       hpc test-sle     sjf4  R       6:48      1 n2154                  273
             11684       hpc test-sle     sjf4  R       6:48      1 n2012                  273
             11685       hpc test-sle     sjf4  R       6:48      1 n2071                  273
             11686       hpc test-sle     sjf4  R       6:48      1 n2094                  273

JobId=11690 JobName=test-sleep
   UserId=hyaktest(383794) GroupId=hyak-hpc(202025) MCS_label=N/A
   Priority=275 Nice=0 Account=hpc QOS=normal
   JobState=PENDING Reason=QOSGrpCpuLimit Dependency=(null)
   Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
   RunTime=00:00:00 TimeLimit=05:00:00 TimeMin=N/A
   SubmitTime=2017-05-05T13:19:22 EligibleTime=2017-05-05T13:19:22
   StartTime=Unknown EndTime=Unknown Deadline=N/A
   PreemptTime=None SuspendTime=None SecsPreSuspend=0
   Partition=hpc AllocNode:Sid=mgmt4:53509
   ReqNodeList=(null) ExcNodeList=(null)
   NodeList=(null) SchedNodeList=n[2002,2102-2106]
   NumNodes=6-6 NumCPUs=6 NumTasks=6 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
   TRES=cpu=6,mem=614400,node=6
   Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
   MinCPUsNode=1 MinMemoryNode=100G MinTmpDiskNode=0
   Features=(null) DelayBoot=00:00:00
   Gres=(null) Reservation=(null)
   OverSubscribe=NO Contiguous=0 Licenses=(null) Network=(null)
   Command=/gscratch/home/hyaktest/test-sleep.sh
   WorkDir=/home/hyaktest
   StdErr=/home/hyaktest/slurm-11690.out
   StdIn=/dev/null
   StdOut=/home/hyaktest/slurm-11690.out
   Power=


The real world set of jobs that set me on this investigation are in the bkrs partition. Here's a subset of that display.

             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)             PRIORITY
             11103      bkrs magnetic     mjs5 PD       0:00     32 (QOSGrpCpuLimit)                  293
             11033      bkrs   piping wdetmold PD       0:00      1 (QOSGrpCpuLimit)                  200
             11034      bkrs   piping wdetmold PD       0:00      1 (QOSGrpCpuLimit)                  200
             11035      bkrs   piping wdetmold PD       0:00      1 (QOSGrpCpuLimit)                  200
             11036      bkrs   piping wdetmold PD       0:00      1 (QOSGrpCpuLimit)                  200
             11037      bkrs   piping wdetmold PD       0:00      1 (QOSGrpCpuLimit)                  200
             11038      bkrs   piping wdetmold PD       0:00      1 (QOSGrpCpuLimit)                  200
             11039      bkrs   piping wdetmold PD       0:00      1 (QOSGrpCpuLimit)                  200
             11040      bkrs   piping wdetmold PD       0:00      1 (QOSGrpCpuLimit)                  200
             11041      bkrs   piping wdetmold PD       0:00      1 (QOSGrpCpuLimit)                  200
             11042      bkrs   piping wdetmold PD       0:00      1 (QOSGrpCpuLimit)                  200
             11043      bkrs   piping wdetmold PD       0:00      1 (QOSGrpCpuLimit)                  200
             11044      bkrs   piping wdetmold PD       0:00      1 (QOSGrpCpuLimit)                  200
             11045      bkrs   piping wdetmold PD       0:00      1 (QOSGrpCpuLimit)                  200
             11046      bkrs   piping wdetmold PD       0:00      1 (QOSGrpCpuLimit)                  200
             11047      bkrs   piping wdetmold PD       0:00      1 (QOSGrpCpuLimit)                  200
             11048      bkrs   piping wdetmold PD       0:00      1 (QOSGrpCpuLimit)                  200
             11049      bkrs   piping wdetmold PD       0:00      1 (QOSGrpCpuLimit)                  200
             11050      bkrs   piping wdetmold PD       0:00      1 (QOSGrpCpuLimit)                  200
             11032      bkrs   piping wdetmold  R      15:41      1 n2119                  200
             11031      bkrs   piping wdetmold  R      43:13      1 n2142                  200
             11030      bkrs   piping wdetmold  R    1:07:26      1 n2149                  200
             11029      bkrs   piping wdetmold  R    1:08:57      1 n2164                  200
             11028      bkrs   piping wdetmold  R    1:22:12      1 n2108                  200
             11027      bkrs   piping wdetmold  R    1:30:42      1 n2179                  200
             11026      bkrs   piping wdetmold  R    1:38:43      1 n2178                  200
             11025      bkrs   piping wdetmold  R    2:09:43      1 n2005                  200
             11024      bkrs   piping wdetmold  R    3:08:55      1 n2080                  200
             11023      bkrs   piping wdetmold  R    3:11:55      1 n2161                  200
             11022      bkrs   piping wdetmold  R    3:16:13      1 n2054                  200
             11021      bkrs   piping wdetmold  R    3:29:43      1 n2077                  200
             11019      bkrs   piping wdetmold  R    3:38:56      1 n2159                  200


JobId=11103 JobName=magnetic
   UserId=mjs5(261746) GroupId=hyak-bkrs(20723) MCS_label=N/A
   Priority=293 Nice=0 Account=bkrs QOS=normal
   JobState=PENDING Reason=QOSGrpCpuLimit Dependency=(null)
   Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
   RunTime=00:00:00 TimeLimit=1-00:00:00 TimeMin=N/A
   SubmitTime=2017-05-01T20:37:08 EligibleTime=2017-05-01T20:37:08
   StartTime=Unknown EndTime=Unknown Deadline=N/A
   PreemptTime=None SuspendTime=None SecsPreSuspend=0
   Partition=bkrs AllocNode:Sid=mox2:37967
   ReqNodeList=(null) ExcNodeList=(null)
   NodeList=(null) SchedNodeList=n[2012-2013,2019-2021,2033-2035,2062-2064,2070-2072,2083-2085,2102-2106,2114-2116,2130-2133,2180-2182]
   NumNodes=32-32 NumCPUs=512 NumTasks=512 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
   TRES=cpu=512,mem=3932160,node=32
   Socks/Node=* NtasksPerN:B:S:C=16:0:*:* CoreSpec=*
   MinCPUsNode=16 MinMemoryNode=120G MinTmpDiskNode=0
   Features=(null) DelayBoot=00:00:00
   Gres=(null) Reservation=(null)
   OverSubscribe=NO Contiguous=0 Licenses=(null) Network=(null)
   Command=/gscratch/bkrs/Magnetic/cl21_48_96_b6p3_m0p2416_m0p2050_Magnetic/slurm_qqq_mox_Magnetic_160MeV_b6p3_48_96.slurm
   WorkDir=/gscratch/bkrs/Magnetic/cl21_48_96_b6p3_m0p2416_m0p2050_Magnetic
   StdErr=/gscratch/bkrs/Magnetic/cl21_48_96_b6p3_m0p2416_m0p2050_Magnetic/jobfiles/slurm.%N.11103.err
   StdIn=/dev/null
   StdOut=/gscratch/bkrs/Magnetic/cl21_48_96_b6p3_m0p2416_m0p2050_Magnetic/jobfiles/slurm.%N.11103.out
   Power=

I will attach the log with the backfill debugging momentarily.
Comment 6 Stephen Fralich 2017-05-05 14:37:44 MDT
Created attachment 4521 [details]
Slurmctld log with backfill debugging
Comment 7 Dominik Bartkiewicz 2017-05-16 05:35:08 MDT
Hi

17.02.3 should solve all of this issues.

Dominik
Comment 8 Stephen Fralich 2017-05-18 12:56:11 MDT
I updated to 17.02.3, but the behavior appears to be exactly the same. This is a serious bug for my site. Large jobs effectively will never start if there are enough small jobs competing for resources in the same floating partition. We have service level agreements with our customers that the right jobs will start and they'll start within a certain period of time, so we're not presently meeting those requirements.

Can you provide a time frame as to when you can replicate this bug? If you can't, please let me know that so I can provide whatever information is necessary for you to do so. How often should I expect updates on your progress? When should I expect a patch to fix this issue? If this fix is not on your short list, I'm going to have to make other provisions like coming up with an alternate scheduler configuration. This is the one that was recommended to me by the SchedMD staff at SC15 however (bug 3013).

I will attach a new log with backfill debugging and my current configuration (approximately the same as the old one).

Test Procedure:
I start 7 one node jobs (14777-14833) under my user id with a 1 day WC limit.

I start 1 six node job (14485) under hyaktest with a 5 hour WC limit.

I increase the priority of the job started as hyaktest:  scontrol update JobId=14485 nice=-1000

I submit 10 additional jobs (14486-14495) under my userid with a 3 day WC limit.

The additional jobs start via backfill even though this will prevent hyaktest's job from starting:
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON) PRIORITY
             14485       hpc test-sle hyaktest PD       0:00      6 (QOSGrpCpuLimit) 1279
             14491       hpc test-sle     sjf4 PD       0:00      1 (Priority) 278
             14492       hpc test-sle     sjf4 PD       0:00      1 (Priority) 278
             14493       hpc test-sle     sjf4 PD       0:00      1 (Priority) 278
             14494       hpc test-sle     sjf4 PD       0:00      1 (Priority) 278
             14495       hpc test-sle     sjf4 PD       0:00      1 (Priority) 278
             14486       hpc test-sle     sjf4  R       0:55      1 n2172 278
             14487       hpc test-sle     sjf4  R       0:55      1 n2043 278
             14488       hpc test-sle     sjf4  R       0:55      1 n2048 278
             14489       hpc test-sle     sjf4  R       0:55      1 n2053 278
             14490       hpc test-sle     sjf4  R       0:55      1 n2056 278
             14479       hpc test-sle     sjf4  R      11:54      1 n2149 278
             14480       hpc test-sle     sjf4  R      11:54      1 n2150 278
             14481       hpc test-sle     sjf4  R      11:54      1 n2001 278
             14482       hpc test-sle     sjf4  R      11:54      1 n2003 278
             14483       hpc test-sle     sjf4  R      11:54      1 n2009 278
             14477       hpc test-sle     sjf4  R      11:57      1 n2173 278
             14478       hpc test-sle     sjf4  R      11:57      1 n2148 278


I cancel 4 jobs (14477-14480) running under my userid with a 1 day WC limit.

The additional jobs start even though again this will prevent the hyaktest job from starting:
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON) PRIORITY
             14485       hpc test-sle hyaktest PD       0:00      6 (QOSGrpCpuLimit) 1279
             14495       hpc test-sle     sjf4 PD       0:00      1 (Priority) 278
             14491       hpc test-sle     sjf4  R       0:17      1 n2173 278
             14492       hpc test-sle     sjf4  R       0:17      1 n2024 278
             14493       hpc test-sle     sjf4  R       0:17      1 n2026 278
             14494       hpc test-sle     sjf4  R       0:17      1 n2028 278
             14486       hpc test-sle     sjf4  R       2:48      1 n2172 278
             14487       hpc test-sle     sjf4  R       2:48      1 n2043 278
             14488       hpc test-sle     sjf4  R       2:48      1 n2048 278
             14489       hpc test-sle     sjf4  R       2:48      1 n2053 278
             14490       hpc test-sle     sjf4  R       2:48      1 n2056 278
             14481       hpc test-sle     sjf4  R      13:47      1 n2001 278
             14482       hpc test-sle     sjf4  R      13:47      1 n2003 278
             14483       hpc test-sle     sjf4  R      13:47      1 n2009 278
Comment 9 Stephen Fralich 2017-05-18 12:58:02 MDT
Created attachment 4587 [details]
Slurmctld 17023 log with backfill debugging
Comment 10 Stephen Fralich 2017-05-18 12:58:42 MDT
Created attachment 4588 [details]
Slurm 17023 configuration
Comment 11 Dominik Bartkiewicz 2017-05-19 09:39:33 MDT
Hi

I can reproduce similar behavior.
I can't give you time frames but
I will try to attach some preliminary patch to the end of the day.

Dominik
Comment 16 Stephen Fralich 2017-05-19 11:54:26 MDT
Thank you. I appreciate knowing you've replicated the issue and that you're working on a new patch. Having something to pass back to customers experiencing this issue is important.
Comment 20 Dominik Bartkiewicz 2017-05-19 14:40:49 MDT
Created attachment 4599 [details]
preliminary patch

Hi

This patch should help in this problem.
But it has some limitations, it works only for partition qos group limits.
General solution is more complicated and we need more time to prepare this.

Dominik
Comment 21 Stephen Fralich 2017-05-19 16:40:10 MDT
Ok. I understand. Thanks for the quick patch. I'll put this in early next week and let you know how it goes.
Comment 24 Stephen Fralich 2017-05-22 17:08:54 MDT
The patch does not help the test case I've outlined. I see there's a new message for backfill debugging in the patch which indicates the code has done something, but we're not seeing that message. I'm not sure what the difference could be between our test setups, but I feel like I've outlined everything relevant here. I double checked to make sure I applied the patch correctly and I have. I'll attach the log from my latest test (jobs running in the hpc partition are the relevant ones).
Comment 25 Stephen Fralich 2017-05-22 17:09:38 MDT
Created attachment 4612 [details]
Slurmctld backfill debugging log after initial patch
Comment 26 Dominik Bartkiewicz 2017-05-22 17:35:39 MDT
Dominik

Could you send me: "scontrol show job" output?

Dominik
Comment 27 Stephen Fralich 2017-05-22 17:49:50 MDT
**** State after submitting single node jobs (12 hrs WC), submitting multi-node job (5 hrs WC), and adjusting priority on multi-node job ****

             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON) PRIORITY
             15352       hpc test-sle hyaktest PD       0:00      6 (QOSGrpCpuLimit) 679
             15351       hpc test-sle     sjf4  R       2:37      1 n2145 278
             15346       hpc test-sle     sjf4  R       2:40      1 n2157 278
             15347       hpc test-sle     sjf4  R       2:40      1 n2141 278
             15348       hpc test-sle     sjf4  R       2:40      1 n2142 278
             15349       hpc test-sle     sjf4  R       2:40      1 n2143 278
             15350       hpc test-sle     sjf4  R       2:40      1 n2144 278
             15345       hpc test-sle     sjf4  R       2:43      1 n2138 278

JobId=15352 JobName=test-sleep
   UserId=hyaktest(383794) GroupId=hyak-hpc(202025) MCS_label=N/A
   Priority=679 Nice=-400 Account=hpc QOS=normal
   JobState=PENDING Reason=QOSGrpCpuLimit Dependency=(null)
   Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
   RunTime=00:00:00 TimeLimit=05:00:00 TimeMin=N/A
   SubmitTime=2017-05-22T16:39:53 EligibleTime=2017-05-22T16:39:53
   StartTime=2017-05-22T16:48:41 EndTime=2017-05-22T21:48:41 Deadline=N/A
   PreemptTime=None SuspendTime=None SecsPreSuspend=0
   Partition=hpc AllocNode:Sid=mgmt4:18604
   ReqNodeList=(null) ExcNodeList=(null)
   NodeList=(null) SchedNodeList=n[2116-2121]
   NumNodes=6-6 NumCPUs=6 NumTasks=6 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
   TRES=cpu=6,mem=614400,node=6
   Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
   MinCPUsNode=1 MinMemoryNode=100G MinTmpDiskNode=0
   Features=(null) DelayBoot=00:00:00
   Gres=(null) Reservation=(null)
   OverSubscribe=NO Contiguous=0 Licenses=(null) Network=(null)
   Command=/gscratch/home/hyaktest/test-sleep.sh
   WorkDir=/home/hyaktest
   StdErr=/home/hyaktest/slurm-15352.out
   StdIn=/dev/null
   StdOut=/home/hyaktest/slurm-15352.out
   Power=


**** State after submitting additional single node jobs (48 hrs WC) and some are backfilled ****

             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON) PRIORITY
             15352       hpc test-sle hyaktest PD       0:00      6 (QOSGrpCpuLimit) 679
             15359       hpc test-sle     sjf4 PD       0:00      1 (QOSGrpCpuLimit) 278
             15360       hpc test-sle     sjf4 PD       0:00      1 (QOSGrpCpuLimit) 278
             15361       hpc test-sle     sjf4 PD       0:00      1 (QOSGrpCpuLimit) 278
             15362       hpc test-sle     sjf4 PD       0:00      1 (QOSGrpCpuLimit) 278
             15354       hpc test-sle     sjf4  R       0:04      1 n2146 278
             15355       hpc test-sle     sjf4  R       0:04      1 n2147 278
             15356       hpc test-sle     sjf4  R       0:04      1 n2167 278
             15357       hpc test-sle     sjf4  R       0:04      1 n2168 278
             15358       hpc test-sle     sjf4  R       0:04      1 n2169 278
             15351       hpc test-sle     sjf4  R       5:07      1 n2145 278
             15346       hpc test-sle     sjf4  R       5:10      1 n2157 278
             15347       hpc test-sle     sjf4  R       5:10      1 n2141 278
             15348       hpc test-sle     sjf4  R       5:10      1 n2142 278
             15349       hpc test-sle     sjf4  R       5:10      1 n2143 278
             15350       hpc test-sle     sjf4  R       5:10      1 n2144 278
             15345       hpc test-sle     sjf4  R       5:13      1 n2138 278

JobId=15352 JobName=test-sleep
   UserId=hyaktest(383794) GroupId=hyak-hpc(202025) MCS_label=N/A
   Priority=679 Nice=-400 Account=hpc QOS=normal
   JobState=PENDING Reason=QOSGrpCpuLimit Dependency=(null)
   Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
   RunTime=00:00:00 TimeLimit=05:00:00 TimeMin=N/A
   SubmitTime=2017-05-22T16:39:53 EligibleTime=2017-05-22T16:39:53
   StartTime=2017-05-22T16:52:46 EndTime=2017-05-22T21:52:46 Deadline=N/A
   PreemptTime=None SuspendTime=None SecsPreSuspend=0
   Partition=hpc AllocNode:Sid=mgmt4:18604
   ReqNodeList=(null) ExcNodeList=(null)
   NodeList=(null) SchedNodeList=n[2116-2121]
   NumNodes=6-6 NumCPUs=6 NumTasks=6 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
   TRES=cpu=6,mem=614400,node=6
   Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
   MinCPUsNode=1 MinMemoryNode=100G MinTmpDiskNode=0
   Features=(null) DelayBoot=00:00:00
   Gres=(null) Reservation=(null)
   OverSubscribe=NO Contiguous=0 Licenses=(null) Network=(null)
   Command=/gscratch/home/hyaktest/test-sleep.sh
   WorkDir=/home/hyaktest
   StdErr=/home/hyaktest/slurm-15352.out
   StdIn=/dev/null
   StdOut=/home/hyaktest/slurm-15352.out
   Power=

  Job ID    Username    Nodes    Cores    Memory    Time Remaining
--------    --------    -----    -----    ------    --------------
   15345        sjf4        1       28      100G          11:53:03
   15346        sjf4        1       28      100G          11:53:06
   15347        sjf4        1       28      100G          11:53:06
   15348        sjf4        1       28      100G          11:53:06
   15349        sjf4        1       28      100G          11:53:06
   15350        sjf4        1       28      100G          11:53:06
   15351        sjf4        1       28      100G          11:53:09
   15354        sjf4        1       28      100G        1-23:58:12
   15355        sjf4        1       28      100G        1-23:58:12
   15356        sjf4        1       28      100G        1-23:58:12
   15357        sjf4        1       28      100G        1-23:58:12
   15358        sjf4        1       28      100G        1-23:58:12

**** Cancelled 2 hr WC single node jobs and the waiting ones started *****

             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON) PRIORITY
             15352       hpc test-sle hyaktest PD       0:00      6 (QOSGrpCpuLimit) 679
             15359       hpc test-sle     sjf4  R       0:01      1 n2146 278
             15360       hpc test-sle     sjf4  R       0:01      1 n2147 278
             15361       hpc test-sle     sjf4  R       0:01      1 n2167 278
             15362       hpc test-sle     sjf4  R       0:01      1 n2168 278
             15351       hpc test-sle     sjf4  R       9:04      1 n2145 278
             15346       hpc test-sle     sjf4  R       9:07      1 n2157 278
             15347       hpc test-sle     sjf4  R       9:07      1 n2141 278
             15348       hpc test-sle     sjf4  R       9:07      1 n2142 278
             15349       hpc test-sle     sjf4  R       9:07      1 n2143 278
             15350       hpc test-sle     sjf4  R       9:07      1 n2144 278
             15345       hpc test-sle     sjf4  R       9:10      1 n2138 278

JobId=15352 JobName=test-sleep
   UserId=hyaktest(383794) GroupId=hyak-hpc(202025) MCS_label=N/A
   Priority=679 Nice=-400 Account=hpc QOS=normal
   JobState=PENDING Reason=QOSGrpCpuLimit Dependency=(null)
   Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
   RunTime=00:00:00 TimeLimit=05:00:00 TimeMin=N/A
   SubmitTime=2017-05-22T16:39:53 EligibleTime=2017-05-22T16:39:53
   StartTime=2017-05-22T16:57:16 EndTime=2017-05-22T21:57:16 Deadline=N/A
   PreemptTime=None SuspendTime=None SecsPreSuspend=0
   Partition=hpc AllocNode:Sid=mgmt4:18604
   ReqNodeList=(null) ExcNodeList=(null)
   NodeList=(null) SchedNodeList=n[2116-2121]
   NumNodes=6-6 NumCPUs=6 NumTasks=6 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
   TRES=cpu=6,mem=614400,node=6
   Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
   MinCPUsNode=1 MinMemoryNode=100G MinTmpDiskNode=0
   Features=(null) DelayBoot=00:00:00
   Gres=(null) Reservation=(null)
   OverSubscribe=NO Contiguous=0 Licenses=(null) Network=(null)
   Command=/gscratch/home/hyaktest/test-sleep.sh
   WorkDir=/home/hyaktest
   StdErr=/home/hyaktest/slurm-15352.out
   StdIn=/dev/null
   StdOut=/home/hyaktest/slurm-15352.out
   Power=

  Job ID    Username    Nodes    Cores    Memory    Time Remaining
--------    --------    -----    -----    ------    --------------
   15345        sjf4        1       28      100G          11:49:48
   15346        sjf4        1       28      100G          11:49:51
   15347        sjf4        1       28      100G          11:49:51
   15348        sjf4        1       28      100G          11:49:51
   15349        sjf4        1       28      100G          11:49:51
   15350        sjf4        1       28      100G          11:49:51
   15351        sjf4        1       28      100G          11:49:54
   15359        sjf4        1       28      100G        1-23:58:57
   15360        sjf4        1       28      100G        1-23:58:57
   15361        sjf4        1       28      100G        1-23:58:57
   15362        sjf4        1       28      100G        1-23:58:57
Comment 28 Stephen Fralich 2017-05-22 17:53:14 MDT
Created attachment 4613 [details]
Patched slurmctld showing log when scontrol output taken
Comment 29 Stephen Fralich 2017-05-22 17:57:38 MDT
After thinking about it, I figured out you probably wanted all of the output for every job. I've repeated that test and I'll attach that output.

             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON) PRIORITY
             15373       hpc test-sle hyaktest PD       0:00      6 (QOSGrpCpuLimit) 679
             15379       hpc test-sle     sjf4 PD       0:00      1 (Priority) 278
             15380       hpc test-sle     sjf4 PD       0:00      1 (Priority) 278
             15381       hpc test-sle     sjf4 PD       0:00      1 (Priority) 278
             15374       hpc test-sle     sjf4  R       0:42      1 n2150 278
             15375       hpc test-sle     sjf4  R       0:42      1 n2146 278
             15376       hpc test-sle     sjf4  R       0:42      1 n2147 278
             15377       hpc test-sle     sjf4  R       0:42      1 n2167 278
             15378       hpc test-sle     sjf4  R       0:42      1 n2168 278
             15368       hpc test-sle     sjf4  R       2:02      1 n2141 278
             15369       hpc test-sle     sjf4  R       2:02      1 n2142 278
             15370       hpc test-sle     sjf4  R       2:02      1 n2143 278
             15371       hpc test-sle     sjf4  R       2:02      1 n2144 278
             15372       hpc test-sle     sjf4  R       2:02      1 n2145 278
             15366       hpc test-sle     sjf4  R       2:05      1 n2138 278
             15367       hpc test-sle     sjf4  R       2:05      1 n2157 278
Comment 30 Stephen Fralich 2017-05-22 17:58:44 MDT
Created attachment 4614 [details]
scontrol show job output
Comment 31 Stephen Fralich 2017-05-22 18:03:44 MDT
Created attachment 4615 [details]
slurmctld log corresponding to scontrol show job
Comment 32 Dominik Bartkiewicz 2017-05-23 04:15:59 MDT
Hi

Do you still use SchedulerParameters assoc_limit_stop?

Dominik
Comment 33 Stephen Fralich 2017-05-23 08:38:41 MDT
Yes, the configuration attached to the bug is the one we use.

On May 23, 2017 3:16 AM, <bugs@schedmd.com> wrote:

*Comment # 32 <https://bugs.schedmd.com/show_bug.cgi?id=3776#c32> on bug
3776 <https://bugs.schedmd.com/show_bug.cgi?id=3776> from Dominik
Bartkiewicz <bart@schedmd.com> *

Hi

Do you still use SchedulerParameters assoc_limit_stop?

Dominik

------------------------------
You are receiving this mail because:

   - You reported the bug.
Comment 36 Dominik Bartkiewicz 2017-05-24 06:40:43 MDT
Hi

Sorry for inconvenience.
I am surprised that this patch didn't help.
I was sure that I have fully recreated this locally.
We prepared new patch which is more completed solution but with similar logic. 
I have found some mistakes in current patch witch can cause it doesn't work properly.

Could you try with new patch and with enable debug flag 
Backfill and setdebug  5 and attach a chunk of the logs?

Turn on:
scontrol setdebugflag +backfill
scontrol setdebug 5
Turn off:
scontrol setdebugflag -backfill
scontrol setdebug 2

Dominik
Comment 37 Dominik Bartkiewicz 2017-05-24 08:18:27 MDT
Sorry, I forget to mention that "patch proposal v3" can be only applied to current git source from branch "slurm-17.02". (git checkout  slurm-17.02)

to be sure:
- partition hpc uses qos hpc.


Could you send me output from "scontrol show asso" and "sacctmgr show qos"?

Dominik
Comment 38 Stephen Fralich 2017-05-24 10:36:54 MDT
Created attachment 4635 [details]
show asso output
Comment 39 Stephen Fralich 2017-05-24 10:37:14 MDT
Created attachment 4636 [details]
show qos output
Comment 40 Stephen Fralich 2017-05-24 10:39:23 MDT
I attached that output you requested.

I'll build the updates today and install them tomorrow morning.

When you say:

> to be sure:
> - partition hpc uses qos hpc.

This is what you mean?

PartitionName=hpc Nodes=n2[001-184,195-202,206-210] Default=NO AllowAccounts=hpc QOS=hpc Priority=1000 DefaultTime=1:00:00 OverSubscribe=EXCLUSIVE PreemptMode=OFF

That's how I have it set.
Comment 44 Stephen Fralich 2017-05-25 16:44:16 MDT
This patch looks like it's fixed the issue. I'll attach the section of the logs with that debugging information shortly. Is this fix expected to show up in 17.02.4? Thanks a lot.

             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON) PRIORITY
             16842       hpc test-sle hyaktest PD       0:00      6 (QOSGrpCpuLimit) 680
             16843       hpc test-sle     sjf4 PD       0:00      1 (Priority) 278
             16844       hpc test-sle     sjf4 PD       0:00      1 (Priority) 278
             16845       hpc test-sle     sjf4 PD       0:00      1 (Priority) 278
             16846       hpc test-sle     sjf4 PD       0:00      1 (Priority) 278
             16847       hpc test-sle     sjf4 PD       0:00      1 (Priority) 278
             16848       hpc test-sle     sjf4 PD       0:00      1 (Priority) 278
             16849       hpc test-sle     sjf4 PD       0:00      1 (Priority) 278
             16850       hpc test-sle     sjf4 PD       0:00      1 (Priority) 278
             16851       hpc test-sle     sjf4 PD       0:00      1 (Priority) 278
             16837       hpc test-sle     sjf4  R       2:15      1 n2144 278
             16838       hpc test-sle     sjf4  R       2:15      1 n2145 278
             16839       hpc test-sle     sjf4  R       2:15      1 n2010 278
             16840       hpc test-sle     sjf4  R       2:15      1 n2024 278
             16841       hpc test-sle     sjf4  R       2:15      1 n2039 278
             16835       hpc test-sle     sjf4  R       2:17      1 n2172 278
             16836       hpc test-sle     sjf4  R       2:17      1 n2143 278
             16834       hpc test-sle     sjf4  R       2:20      1 n2171 278
Comment 45 Stephen Fralich 2017-05-25 16:44:51 MDT
Created attachment 4652 [details]
slurmctld log after patch v3
Comment 46 Dominik Bartkiewicz 2017-05-26 06:46:24 MDT
Hi

I am glad to hear that patch helped.
It will be in 17.02.4.
I will let you know when we add it to repo.

Dominik
Comment 51 Dominik Bartkiewicz 2017-05-30 10:45:59 MDT
Hi

Now this fix with additional patches is in repo and will be included in 17.02.4.
Thanks for patience

Dominik