| Summary: | Partition QOS Prevents Legitimate Jobs from Starting | ||
|---|---|---|---|
| Product: | Slurm | Reporter: | Stephen Fralich <sjf4> |
| Component: | Scheduling | Assignee: | Dominik Bartkiewicz <dbartkiewicz> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | 2 - High Impact | ||
| Priority: | --- | CC: | ahough |
| Version: | 17.02.3 | ||
| Hardware: | Linux | ||
| OS: | Linux | ||
| Site: | University of Washington | Slinky Site: | --- |
| Alineos Sites: | --- | Atos/Eviden Sites: | --- |
| Confidential Site: | --- | Coreweave sites: | --- |
| Cray Sites: | --- | DS9 clusters: | --- |
| Google sites: | --- | HPCnow Sites: | --- |
| HPE Sites: | --- | IBM Sites: | --- |
| NOAA SIte: | --- | NoveTech Sites: | --- |
| Nvidia HWinf-CS Sites: | --- | OCF Sites: | --- |
| Recursion Pharma Sites: | --- | SFW Sites: | --- |
| SNIC sites: | --- | Tzag Elita Sites: | --- |
| Linux Distro: | --- | Machine Name: | |
| CLE Version: | Version Fixed: | 17.02.4 | |
| Target Release: | --- | DevPrio: | --- |
| Emory-Cloud Sites: | --- | ||
| Attachments: |
scontrol show asso output
slurm.conf sinfo output sacctmgr show qos output scontrol show job output slurmctld log patch |
||
|
Description
Stephen Fralich
2017-06-01 16:11:19 MDT
Created attachment 4675 [details]
scontrol show asso output
Created attachment 4676 [details]
slurm.conf
Created attachment 4677 [details]
sinfo output
Created attachment 4678 [details]
sacctmgr show qos output
Created attachment 4679 [details]
scontrol show job output
Created attachment 4680 [details]
slurmctld log
That looks like plenty of details to start with. I'm asking Dominik to start looking into this ASAP. It does look like we need to revisit how some of the limits are applied; checking the partition QOS only after freeing up the requisite nodes does appear to lead to some unusual and undesirable side effects, although whether that can be easily solved is something he's much better suited to address. - Tim Hi Which exactly version you use. 17.02.3+patch from 3776 or something from repo? Could you increase debug and attach a chunk of the logs? Turn on: scontrol setdebug 5 Turn off: scontrol setdebug 2 Dominik We're running the slurm-17.02 branch (grabbed on 5/24) with the patch applied from bug 3776. The slurm log that's attached to this bug has the information you want starting at line 906. Created attachment 4683 [details]
patch
Hi
I found bug in previous patch.
This problem is also in current repo version.
This patch should fix this and side effects like preemption problem.
Dominik
Hi This fix is already in repo: https://github.com/SchedMD/slurm/commit/59a820add Dominik Thanks. It sounds like I just need to clone the slurm-17.02 branch then. I'll build that today, install it tomorrow morning, and let you know how that goes. Hi I am going to close this. Today we should release 17.02.4. Please reopen if you have any new issue. Dominik |