Ticket 3859

Summary: Partition QOS Prevents Legitimate Jobs from Starting
Product: Slurm Reporter: Stephen Fralich <sjf4>
Component: SchedulingAssignee: Dominik Bartkiewicz <dbartkiewicz>
Status: RESOLVED FIXED QA Contact:
Severity: 2 - High Impact    
Priority: --- CC: ahough
Version: 17.02.3   
Hardware: Linux   
OS: Linux   
Site: University of Washington Slinky Site: ---
Alineos Sites: --- Atos/Eviden Sites: ---
Confidential Site: --- Coreweave sites: ---
Cray Sites: --- DS9 clusters: ---
Google sites: --- HPCnow Sites: ---
HPE Sites: --- IBM Sites: ---
NOAA SIte: --- NoveTech Sites: ---
Nvidia HWinf-CS Sites: --- OCF Sites: ---
Recursion Pharma Sites: --- SFW Sites: ---
SNIC sites: --- Tzag Elita Sites: ---
Linux Distro: --- Machine Name:
CLE Version: Version Fixed: 17.02.4
Target Release: --- DevPrio: ---
Emory-Cloud Sites: ---
Attachments: scontrol show asso output
slurm.conf
sinfo output
sacctmgr show qos output
scontrol show job output
slurmctld log
patch

Description Stephen Fralich 2017-06-01 16:11:19 MDT
This problem seems to be a complication of fixing bug 3776. All the below jobs blocked from starting are reporting in the debug level log they're blocked by the partition QOS. This group is well below their limit of 40 nodes however.

I'm not sure if it's related or not, but I've noticed that Slrum will preempt jobs to start a job that's blocked by the partition QOS. It doesn't seem like the code that determines whether or not a job should be preempted is partition qos aware. This leaves a lot of nodes idle even though there's more than enough work queued to fill all the nodes.

I've not found a way to cause this condition to happen in our test allocation. I've only observed it in customer allocations. I'll attach our config, logs and various scontrol outputs. Let me know if you need something more.


Allocation Name    Nodes    Cores    Total Memory
---------------    -----    -----    ------------
stf                   40     1120           4880G


  Job ID    Username    Nodes    Cores    Memory    Time Remaining
--------    --------    -----    -----    ------    --------------
   16800        hes8        1       28       30G        1-08:14:06
   16801        hes8        1       28       30G        1-08:14:06
   16802        hes8        1       28       30G        1-08:14:06
   16803        hes8        1       28       30G        1-08:31:30
   18640        ahc9        4      112      800G           5:02:49
   19061    hdnelson        1       28      128G          19:13:06
--------    --------    -----    -----    ------    --------------
                            9      252     1048G


             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
             19741       stf spades_J   cselig PD       0:00      1 (Resources)
             19743       stf spades_6   cselig PD       0:00      1 (Resources)
             19185       stf R5global   vanouk PD       0:00     11 (Resources)
             19754       stf 5x5_shee ldgibson PD       0:00      1 (Resources)
             19444       stf NIMROD_t   morgak PD       0:00      8 (Resources)
             19729       stf  R5local   vanouk PD       0:00      8 (Resources)
             19213       stf gemhires   idatta PD       0:00      4 (Resources)
             19123       stf R5orig.p   vanouk PD       0:00      8 (Resources)
             18640       stf H2-015-m     ahc9  R 3-02:49:33      4 n[2091-2094]
             16803       stf 2l77_mut     hes8  R 6-23:20:52      1 n2173
             16800       stf 2l77_mix     hes8  R 6-23:38:16      1 n2167
             16801       stf 2l77_mut     hes8  R 6-23:38:16      1 n2169
             16802       stf 2l77_pop     hes8  R 6-23:38:16      1 n2170
             19061       stf AIS35_Zn hdnelson  R 2-04:39:16      1 n2133



[2017-06-01T14:12:23.348] backfill test for JobID=19123 Prio=1 Partition=stf
[2017-06-01T14:12:23.348] Part QOS blocked_until move start_res to 1496354940
[2017-06-01T14:12:23.348] Job 19123 to start at 2017-06-01T15:09:00, end at 2017-06-02T07:49:00 on n[2150-2154,2162-2164]

[2017-06-01T14:12:23.348] backfill test for JobID=19213 Prio=1 Partition=stf
[2017-06-01T14:12:23.348] Part QOS blocked_until move start_res to 1496354940
[2017-06-01T14:12:23.348] backfill: Try later job 19213 later_start 1496357700
[2017-06-01T14:12:23.348] Job 19213 to start at 2017-06-01T17:54:40, end at 2017-06-06T17:54:00 on n[2156,2165-2166,2196]

[2017-06-01T14:12:23.350] backfill test for JobID=19729 Prio=1 Partition=stf
[2017-06-01T14:12:23.350] Part QOS blocked_until move start_res to 1496354940
[2017-06-01T14:12:23.351] backfill: Try later job 19729 later_start 1496357700
[2017-06-01T14:12:23.351] backfill: Try later job 19729 later_start 1496364840
[2017-06-01T14:12:23.351] backfill: Try later job 19729 later_start 1496371920
[2017-06-01T14:12:23.352] backfill: Try later job 19729 later_start 1496383740
[2017-06-01T14:12:23.352] backfill: Try later job 19729 later_start 1496414940
[2017-06-01T14:12:23.352] Job 19729 to start at 2017-06-02T07:49:00, end at 2017-06-02T11:09:00 on n[2143-2147,2150-2152]

[2017-06-01T14:12:23.348] backfill test for JobID=19444 Prio=1 Partition=stf
[2017-06-01T14:12:23.348] Part QOS blocked_until move start_res to 1496354940
[2017-06-01T14:12:23.349] backfill: Try later job 19444 later_start 1496357700
[2017-06-01T14:12:23.349] backfill: Try later job 19444 later_start 1496364840
[2017-06-01T14:12:23.349] backfill: Try later job 19444 later_start 1496371920
[2017-06-01T14:12:23.350] backfill: Try later job 19444 later_start 1496383740
[2017-06-01T14:12:23.350] backfill: Try later job 19444 later_start 1496414940
[2017-06-01T14:12:23.350] Job 19444 to start at 2017-06-02T07:49:00, end at 2017-06-03T07:49:00 on n[2134-2139,2162-2163]

[2017-06-01T14:12:23.347] backfill test for JobID=19754 Prio=2 Partition=stf
[2017-06-01T14:12:23.347] Part QOS blocked_until move start_res to 1496354940
[2017-06-01T14:12:23.348] Job 19754 to start at 2017-06-01T15:09:00, end at 2017-06-01T23:09:00 on n2171

[2017-06-01T14:12:23.347] backfill test for JobID=19185 Prio=2 Partition=stf
[2017-06-01T14:12:23.347] Part QOS blocked_until move start_res to 1496354940
[2017-06-01T14:12:23.347] Job 19185 to start at 2017-06-01T15:09:00, end at 2017-06-02T07:49:00 on n[2134-2139,2143-2147]

[2017-06-01T14:12:23.345] backfill test for JobID=19743 Prio=7 Partition=stf
[2017-06-01T14:12:23.345] Part QOS blocked_until move start_res to 1496354940
[2017-06-01T14:12:23.345] backfill: Try later job 19743 later_start 1496357700
[2017-06-01T14:12:23.345] backfill: Try later job 19743 later_start 1496371920
[2017-06-01T14:12:23.345] backfill: Try later job 19743 later_start 1496422680
[2017-06-01T14:12:23.346] backfill: Try later job 19743 later_start 1496422920
[2017-06-01T14:12:23.346] backfill: Job 19743 overlaps with existing reservation start_time=1496520600 end_reserve=1496607000 boot_time=0 later_start 1496520600
[2017-06-01T14:12:23.346] backfill: Try later job 19743 later_start 1496520900
[2017-06-01T14:12:23.346] backfill: Try later job 19743 later_start 1496699340
[2017-06-01T14:12:23.347] backfill: Try later job 19743 later_start 1497046140
[2017-06-01T14:12:23.347] Job 19743 to start at 2017-06-09T15:09:00, end at 2017-06-10T15:09:00 on n2017

[2017-06-01T14:12:23.342] backfill test for JobID=19741 Prio=7 Partition=stf
[2017-06-01T14:12:23.342] Part QOS blocked_until move start_res to 1496354940
[2017-06-01T14:12:23.342] backfill: Try later job 19741 later_start 1496357700
[2017-06-01T14:12:23.343] backfill: Try later job 19741 later_start 1496371920
[2017-06-01T14:12:23.343] backfill: Try later job 19741 later_start 1496422680
[2017-06-01T14:12:23.343] backfill: Try later job 19741 later_start 1496422920
[2017-06-01T14:12:23.343] backfill: Job 19741 overlaps with existing reservation start_time=1496520600 end_reserve=1496607000 boot_time=0 later_start 1496520600
[2017-06-01T14:12:23.344] backfill: Try later job 19741 later_start 1496520900
[2017-06-01T14:12:23.344] backfill: Try later job 19741 later_start 1496699340
[2017-06-01T14:12:23.344] backfill: Try later job 19741 later_start 1497046140
[2017-06-01T14:12:23.344] Job 19741 to start at 2017-06-09T15:09:00, end at 2017-06-10T15:09:00 on n2009
Comment 1 Stephen Fralich 2017-06-01 16:12:06 MDT
Created attachment 4675 [details]
scontrol show asso output
Comment 2 Stephen Fralich 2017-06-01 16:12:25 MDT
Created attachment 4676 [details]
slurm.conf
Comment 3 Stephen Fralich 2017-06-01 16:12:42 MDT
Created attachment 4677 [details]
sinfo output
Comment 4 Stephen Fralich 2017-06-01 16:13:03 MDT
Created attachment 4678 [details]
sacctmgr show qos output
Comment 5 Stephen Fralich 2017-06-01 16:13:24 MDT
Created attachment 4679 [details]
scontrol show job output
Comment 6 Stephen Fralich 2017-06-01 16:13:47 MDT
Created attachment 4680 [details]
slurmctld log
Comment 7 Tim Wickberg 2017-06-01 16:23:48 MDT
That looks like plenty of details to start with. I'm asking Dominik to start looking into this ASAP.

It does look like we need to revisit how some of the limits are applied; checking the partition QOS only after freeing up the requisite nodes does appear to lead to some unusual and undesirable side effects, although whether that can be easily solved is something he's much better suited to address.

- Tim
Comment 8 Dominik Bartkiewicz 2017-06-01 17:26:11 MDT
Hi

Which exactly version you use.
17.02.3+patch from 3776 or something from repo?

Could you increase debug and attach a chunk of the logs?

Turn on:
scontrol setdebug 5
Turn off:
scontrol setdebug 2

Dominik
Comment 9 Stephen Fralich 2017-06-01 17:40:31 MDT
We're running the slurm-17.02 branch (grabbed on 5/24) with the patch applied from bug 3776.

The slurm log that's attached to this bug has the information you want starting at line 906.
Comment 10 Dominik Bartkiewicz 2017-06-02 04:02:41 MDT
Created attachment 4683 [details]
patch

Hi

I found bug in previous patch.
This problem is also in current repo version.
This patch should fix this and side effects like preemption problem.

Dominik
Comment 12 Dominik Bartkiewicz 2017-06-02 07:07:56 MDT
Hi

This fix is already in repo:
https://github.com/SchedMD/slurm/commit/59a820add

Dominik
Comment 13 Stephen Fralich 2017-06-02 11:38:33 MDT
Thanks. It sounds like I just need to clone the slurm-17.02 branch then. I'll build that today, install it tomorrow morning, and let you know how that goes.
Comment 14 Dominik Bartkiewicz 2017-06-06 10:01:47 MDT
Hi

I am going to close this.
Today we should  release 17.02.4.
Please reopen if you have any new issue.

Dominik