This problem seems to be a complication of fixing bug 3776. All the below jobs blocked from starting are reporting in the debug level log they're blocked by the partition QOS. This group is well below their limit of 40 nodes however. I'm not sure if it's related or not, but I've noticed that Slrum will preempt jobs to start a job that's blocked by the partition QOS. It doesn't seem like the code that determines whether or not a job should be preempted is partition qos aware. This leaves a lot of nodes idle even though there's more than enough work queued to fill all the nodes. I've not found a way to cause this condition to happen in our test allocation. I've only observed it in customer allocations. I'll attach our config, logs and various scontrol outputs. Let me know if you need something more. Allocation Name Nodes Cores Total Memory --------------- ----- ----- ------------ stf 40 1120 4880G Job ID Username Nodes Cores Memory Time Remaining -------- -------- ----- ----- ------ -------------- 16800 hes8 1 28 30G 1-08:14:06 16801 hes8 1 28 30G 1-08:14:06 16802 hes8 1 28 30G 1-08:14:06 16803 hes8 1 28 30G 1-08:31:30 18640 ahc9 4 112 800G 5:02:49 19061 hdnelson 1 28 128G 19:13:06 -------- -------- ----- ----- ------ -------------- 9 252 1048G JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 19741 stf spades_J cselig PD 0:00 1 (Resources) 19743 stf spades_6 cselig PD 0:00 1 (Resources) 19185 stf R5global vanouk PD 0:00 11 (Resources) 19754 stf 5x5_shee ldgibson PD 0:00 1 (Resources) 19444 stf NIMROD_t morgak PD 0:00 8 (Resources) 19729 stf R5local vanouk PD 0:00 8 (Resources) 19213 stf gemhires idatta PD 0:00 4 (Resources) 19123 stf R5orig.p vanouk PD 0:00 8 (Resources) 18640 stf H2-015-m ahc9 R 3-02:49:33 4 n[2091-2094] 16803 stf 2l77_mut hes8 R 6-23:20:52 1 n2173 16800 stf 2l77_mix hes8 R 6-23:38:16 1 n2167 16801 stf 2l77_mut hes8 R 6-23:38:16 1 n2169 16802 stf 2l77_pop hes8 R 6-23:38:16 1 n2170 19061 stf AIS35_Zn hdnelson R 2-04:39:16 1 n2133 [2017-06-01T14:12:23.348] backfill test for JobID=19123 Prio=1 Partition=stf [2017-06-01T14:12:23.348] Part QOS blocked_until move start_res to 1496354940 [2017-06-01T14:12:23.348] Job 19123 to start at 2017-06-01T15:09:00, end at 2017-06-02T07:49:00 on n[2150-2154,2162-2164] [2017-06-01T14:12:23.348] backfill test for JobID=19213 Prio=1 Partition=stf [2017-06-01T14:12:23.348] Part QOS blocked_until move start_res to 1496354940 [2017-06-01T14:12:23.348] backfill: Try later job 19213 later_start 1496357700 [2017-06-01T14:12:23.348] Job 19213 to start at 2017-06-01T17:54:40, end at 2017-06-06T17:54:00 on n[2156,2165-2166,2196] [2017-06-01T14:12:23.350] backfill test for JobID=19729 Prio=1 Partition=stf [2017-06-01T14:12:23.350] Part QOS blocked_until move start_res to 1496354940 [2017-06-01T14:12:23.351] backfill: Try later job 19729 later_start 1496357700 [2017-06-01T14:12:23.351] backfill: Try later job 19729 later_start 1496364840 [2017-06-01T14:12:23.351] backfill: Try later job 19729 later_start 1496371920 [2017-06-01T14:12:23.352] backfill: Try later job 19729 later_start 1496383740 [2017-06-01T14:12:23.352] backfill: Try later job 19729 later_start 1496414940 [2017-06-01T14:12:23.352] Job 19729 to start at 2017-06-02T07:49:00, end at 2017-06-02T11:09:00 on n[2143-2147,2150-2152] [2017-06-01T14:12:23.348] backfill test for JobID=19444 Prio=1 Partition=stf [2017-06-01T14:12:23.348] Part QOS blocked_until move start_res to 1496354940 [2017-06-01T14:12:23.349] backfill: Try later job 19444 later_start 1496357700 [2017-06-01T14:12:23.349] backfill: Try later job 19444 later_start 1496364840 [2017-06-01T14:12:23.349] backfill: Try later job 19444 later_start 1496371920 [2017-06-01T14:12:23.350] backfill: Try later job 19444 later_start 1496383740 [2017-06-01T14:12:23.350] backfill: Try later job 19444 later_start 1496414940 [2017-06-01T14:12:23.350] Job 19444 to start at 2017-06-02T07:49:00, end at 2017-06-03T07:49:00 on n[2134-2139,2162-2163] [2017-06-01T14:12:23.347] backfill test for JobID=19754 Prio=2 Partition=stf [2017-06-01T14:12:23.347] Part QOS blocked_until move start_res to 1496354940 [2017-06-01T14:12:23.348] Job 19754 to start at 2017-06-01T15:09:00, end at 2017-06-01T23:09:00 on n2171 [2017-06-01T14:12:23.347] backfill test for JobID=19185 Prio=2 Partition=stf [2017-06-01T14:12:23.347] Part QOS blocked_until move start_res to 1496354940 [2017-06-01T14:12:23.347] Job 19185 to start at 2017-06-01T15:09:00, end at 2017-06-02T07:49:00 on n[2134-2139,2143-2147] [2017-06-01T14:12:23.345] backfill test for JobID=19743 Prio=7 Partition=stf [2017-06-01T14:12:23.345] Part QOS blocked_until move start_res to 1496354940 [2017-06-01T14:12:23.345] backfill: Try later job 19743 later_start 1496357700 [2017-06-01T14:12:23.345] backfill: Try later job 19743 later_start 1496371920 [2017-06-01T14:12:23.345] backfill: Try later job 19743 later_start 1496422680 [2017-06-01T14:12:23.346] backfill: Try later job 19743 later_start 1496422920 [2017-06-01T14:12:23.346] backfill: Job 19743 overlaps with existing reservation start_time=1496520600 end_reserve=1496607000 boot_time=0 later_start 1496520600 [2017-06-01T14:12:23.346] backfill: Try later job 19743 later_start 1496520900 [2017-06-01T14:12:23.346] backfill: Try later job 19743 later_start 1496699340 [2017-06-01T14:12:23.347] backfill: Try later job 19743 later_start 1497046140 [2017-06-01T14:12:23.347] Job 19743 to start at 2017-06-09T15:09:00, end at 2017-06-10T15:09:00 on n2017 [2017-06-01T14:12:23.342] backfill test for JobID=19741 Prio=7 Partition=stf [2017-06-01T14:12:23.342] Part QOS blocked_until move start_res to 1496354940 [2017-06-01T14:12:23.342] backfill: Try later job 19741 later_start 1496357700 [2017-06-01T14:12:23.343] backfill: Try later job 19741 later_start 1496371920 [2017-06-01T14:12:23.343] backfill: Try later job 19741 later_start 1496422680 [2017-06-01T14:12:23.343] backfill: Try later job 19741 later_start 1496422920 [2017-06-01T14:12:23.343] backfill: Job 19741 overlaps with existing reservation start_time=1496520600 end_reserve=1496607000 boot_time=0 later_start 1496520600 [2017-06-01T14:12:23.344] backfill: Try later job 19741 later_start 1496520900 [2017-06-01T14:12:23.344] backfill: Try later job 19741 later_start 1496699340 [2017-06-01T14:12:23.344] backfill: Try later job 19741 later_start 1497046140 [2017-06-01T14:12:23.344] Job 19741 to start at 2017-06-09T15:09:00, end at 2017-06-10T15:09:00 on n2009
Created attachment 4675 [details] scontrol show asso output
Created attachment 4676 [details] slurm.conf
Created attachment 4677 [details] sinfo output
Created attachment 4678 [details] sacctmgr show qos output
Created attachment 4679 [details] scontrol show job output
Created attachment 4680 [details] slurmctld log
That looks like plenty of details to start with. I'm asking Dominik to start looking into this ASAP. It does look like we need to revisit how some of the limits are applied; checking the partition QOS only after freeing up the requisite nodes does appear to lead to some unusual and undesirable side effects, although whether that can be easily solved is something he's much better suited to address. - Tim
Hi Which exactly version you use. 17.02.3+patch from 3776 or something from repo? Could you increase debug and attach a chunk of the logs? Turn on: scontrol setdebug 5 Turn off: scontrol setdebug 2 Dominik
We're running the slurm-17.02 branch (grabbed on 5/24) with the patch applied from bug 3776. The slurm log that's attached to this bug has the information you want starting at line 906.
Created attachment 4683 [details] patch Hi I found bug in previous patch. This problem is also in current repo version. This patch should fix this and side effects like preemption problem. Dominik
Hi This fix is already in repo: https://github.com/SchedMD/slurm/commit/59a820add Dominik
Thanks. It sounds like I just need to clone the slurm-17.02 branch then. I'll build that today, install it tomorrow morning, and let you know how that goes.
Hi I am going to close this. Today we should release 17.02.4. Please reopen if you have any new issue. Dominik