Ticket 6729 - topo gres count underflow error
Summary: topo gres count underflow error
Status: RESOLVED DUPLICATE of ticket 6370
Alias: None
Product: Slurm
Classification: Unclassified
Component: slurmctld (show other tickets)
Version: 18.08.4
Hardware: Linux Linux
: 4 - Minor Issue
Assignee: Director of Support
QA Contact:
URL:
Depends on:
Blocks:
 
Reported: 2019-03-20 09:38 MDT by Davide Vanzo
Modified: 2019-03-25 14:00 MDT (History)
0 users

See Also:
Site: Vanderbilt
Slinky Site: ---
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
Google sites: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
NoveTech Sites: ---
Nvidia HWinf-CS Sites: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Tzag Elita Sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed:
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---

Attachments
slurmctld log (25.22 MB, application/gzip)
2019-03-20 09:38 MDT, Davide Vanzo 		Details
slurmd log (gpu0010) (2.02 KB, application/gzip)
2019-03-20 09:39 MDT, Davide Vanzo 		Details
Slurm configuration (8.18 KB, text/plain)
2019-03-20 09:39 MDT, Davide Vanzo 		Details
GRES configuration (257 bytes, text/plain)
2019-03-20 09:39 MDT, Davide Vanzo 		Details
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this ticket.
Description Davide Vanzo 2019-03-20 09:38:01 MDT 
Hello there,

Recently we started observing in the slurmctld log error messages like this one:

> error: gres/gpu: job 7149683 dealloc node gpu0014 topo gres count underflow (0 1)
The gres.conf configuration has not changed and the error persists after rebooting either slurmctld and slurmd. No particular errors are evident in the slurmd log except some controller connect failure that may be the result of the changes we are testing in Bug#6639.

Attached you con find our current configuration files and the logs. Please let me know if you need additional information.

Davide
Comment 1 Davide Vanzo 2019-03-20 09:38:36 MDT 
Created attachment 9633 [details]
slurmctld log
Comment 2 Davide Vanzo 2019-03-20 09:39:12 MDT 
Created attachment 9634 [details]
slurmd log (gpu0010)
Comment 3 Davide Vanzo 2019-03-20 09:39:31 MDT 
Created attachment 9635 [details]
Slurm configuration
Comment 4 Davide Vanzo 2019-03-20 09:39:53 MDT 
Created attachment 9636 [details]
GRES configuration
Comment 6 Davide Vanzo 2019-03-20 09:48:01 MDT 
For the reference, here are the details of the two jobs reported in the error.

> JobId=7107167 JobName=titan.batch
>    UserId=khanfm(161909) GroupId=nbody(58698) MCS_label=N/A
>    Priority=10 Nice=0 Account=nbody_acc QOS=nbody_maxwell_acc
>    JobState=RUNNING Reason=None Dependency=(null)
>    Requeue=1 Restarts=1 BatchFlag=1 Reboot=0 ExitCode=0:0
>    DerivedExitCode=0:15
>    RunTime=2-14:35:57 TimeLimit=5-00:00:00 TimeMin=N/A
>    SubmitTime=2019-03-17T20:05:59 EligibleTime=2019-03-17T20:08:00
>    AccrueTime=2019-03-17T20:08:00
>    StartTime=2019-03-17T20:08:52 EndTime=2019-03-22T20:08:52 Deadline=N/A
>    PreemptTime=None SuspendTime=None SecsPreSuspend=0
>    LastSchedEval=2019-03-17T20:08:52
>    Partition=maxwell AllocNode:Sid=gw346:21185
>    ReqNodeList=(null) ExcNodeList=(null)
>    NodeList=gpu[0001-0005,0008-0010]
>    BatchHost=gpu0001
>    NumNodes=8 NumCPUs=96 NumTasks=32 CPUs/Task=3 ReqB:S:C:T=0:0:*:*
>    TRES=cpu=96,mem=800G,node=8,billing=96,gres/gpu=32
>    Socks/Node=* NtasksPerN:B:S:C=4:0:*:* CoreSpec=*
>      Nodes=gpu[0001-0005,0008-0010] CPU_IDs=0-11 Mem=102400 GRES_IDX=gpu(IDX:0-3)
>    MinCPUsNode=12 MinMemoryNode=100G MinTmpDiskNode=0
>    Features=(null) DelayBoot=00:00:00
>    OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
>    Command=/gpfs23/data/nbody/khanfm/phi-GRAPE+GPU/5Mrun/run-3m/titan.batch
>    WorkDir=/gpfs23/data/nbody/khanfm/phi-GRAPE+GPU/5Mrun/run-3m
>    StdErr=/gpfs23/data/nbody/khanfm/phi-GRAPE+GPU/5Mrun/run-3m/run_phi.out
>    StdIn=/dev/null
>    StdOut=/gpfs23/data/nbody/khanfm/phi-GRAPE+GPU/5Mrun/run-3m/run_phi.out
>    Power=
>    TresPerNode=gpu:4

> JobId=7234306 JobName=svd_trainnew1.slurm
>    UserId=luoh3(649317) GroupId=grissom_lab(20461) MCS_label=N/A
>    Priority=1 Nice=0 Account=grissom_lab_acc QOS=grissom_lab_maxwell_acc
>    JobState=RUNNING Reason=None Dependency=(null)
>    Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
>    DerivedExitCode=0:0
>    RunTime=13:21:42 TimeLimit=1-00:10:00 TimeMin=N/A
>    SubmitTime=2019-03-19T21:22:17 EligibleTime=2019-03-19T21:22:17
>    AccrueTime=2019-03-19T21:22:17
>    StartTime=2019-03-19T21:22:59 EndTime=2019-03-20T21:32:59 Deadline=N/A
>    PreemptTime=None SuspendTime=None SecsPreSuspend=0
>    LastSchedEval=2019-03-19T21:22:59
>    Partition=maxwell AllocNode:Sid=gw343:13962
>    ReqNodeList=(null) ExcNodeList=(null)
>    NodeList=gpu0007
>    BatchHost=gpu0007
>    NumNodes=1 NumCPUs=3 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
>    TRES=cpu=3,mem=20G,node=1,billing=3,gres/gpu=2
>    Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
>      Nodes=gpu0007 CPU_IDs=2-3,6 Mem=20480 GRES_IDX=gpu(IDX:1-2)
>    MinCPUsNode=1 MinMemoryNode=20G MinTmpDiskNode=0
>    Features=(null) DelayBoot=00:00:00
>    OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
>    Command=/gpfs22/home/luoh3/code/svd_trainnew1.slurm
>    WorkDir=/gpfs22/home/luoh3/code
>    StdErr=/gpfs22/home/luoh3/code/slurm-7234306.out
>    StdIn=/dev/null
>    StdOut=/gpfs22/home/luoh3/code/slurm-7234306.out
>    Power=
>    TresPerNode=gpu:2
Comment 8 Michael Hinton 2019-03-20 10:16:24 MDT 
Hi Davide,

This error is caused by restarting the slurmctld while a job with multiple gpus allocated to it is running.

This is fixed in Slurm 18.08.6; see bug 6370 for more information. If you still run into this error after upgrading, please reopen this ticket and let us know.

Thanks!
Michael

*** This ticket has been marked as a duplicate of ticket 6370 ***
Comment 9 Davide Vanzo 2019-03-20 13:14:19 MDT 
Michael,

I do not know if this is related to my previous inquiry or not. What I am observing is that jobs requesting multiple GPUs with "--gres-flags=enforce-binding" get GPUs belonging to different PCIe root complexes. For example:

> JobId=7235226 JobName=D1790G-eq
>    UserId=jsmith(90423) GroupId=csb(10005) MCS_label=N/A
>    Priority=1 Nice=0 Account=csb_gpu_acc QOS=csb_gpu_pascal_acc
>    JobState=RUNNING Reason=None Dependency=(null)
>    Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
>    DerivedExitCode=0:0
>    RunTime=14:51:38 TimeLimit=1-16:00:00 TimeMin=N/A
>    SubmitTime=2019-03-19T22:47:22 EligibleTime=2019-03-19T22:47:22
>    AccrueTime=2019-03-19T22:47:22
>    StartTime=2019-03-19T22:47:34 EndTime=2019-03-21T14:47:34 Deadline=N/A
>    PreemptTime=None SuspendTime=None SecsPreSuspend=0
>    LastSchedEval=2019-03-19T22:47:34
>    Partition=pascal AllocNode:Sid=gw341:13528
>    ReqNodeList=(null) ExcNodeList=(null)
>    NodeList=gpu0024
>    BatchHost=gpu0024
>    NumNodes=1 NumCPUs=2 NumTasks=2 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
>    TRES=cpu=2,mem=4G,node=1,billing=2,gres/gpu=2
>    Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
>      Nodes=gpu0024 CPU_IDs=0,4 Mem=4096 GRES_IDX=gpu(IDX:0,2)
>    MinCPUsNode=1 MinMemoryNode=4G MinTmpDiskNode=0
>    Features=(null) DelayBoot=00:00:00
>    OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
>    Command=/dors/csb/home/jsmith/proj/Nav/aMD/5x0m/DMPC/D1790G/eq/./runeq.sb
>    WorkDir=/dors/csb/home/jsmith/proj/Nav/aMD/5x0m/DMPC/D1790G/eq
>    StdErr=/dors/csb/home/jsmith/proj/Nav/aMD/5x0m/DMPC/D1790G/eq/runeq.log
>    StdIn=/dev/null
>    StdOut=/dors/csb/home/jsmith/proj/Nav/aMD/5x0m/DMPC/D1790G/eq/runeq.log
>    Power=
>    GresEnforceBind=Yes
>    TresPerNode=gpu:2

> JobId=7235218 JobName=E1784K-eq
>    UserId=jsmith(90423) GroupId=csb(10005) MCS_label=N/A
>    Priority=1 Nice=0 Account=csb_gpu_acc QOS=csb_gpu_pascal_acc
>    JobState=RUNNING Reason=None Dependency=(null)
>    Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
>    DerivedExitCode=0:0
>    RunTime=14:53:22 TimeLimit=1-16:00:00 TimeMin=N/A
>    SubmitTime=2019-03-19T22:46:29 EligibleTime=2019-03-19T22:46:29
>    AccrueTime=2019-03-19T22:46:29
>    StartTime=2019-03-19T22:47:34 EndTime=2019-03-21T14:47:34 Deadline=N/A
>    PreemptTime=None SuspendTime=None SecsPreSuspend=0
>    LastSchedEval=2019-03-19T22:47:34
>    Partition=pascal AllocNode:Sid=gw341:13528
>    ReqNodeList=(null) ExcNodeList=(null)
>    NodeList=gpu0017
>    BatchHost=gpu0017
>    NumNodes=1 NumCPUs=2 NumTasks=2 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
>    TRES=cpu=2,mem=4G,node=1,billing=2,gres/gpu=2
>    Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
>      Nodes=gpu0017 CPU_IDs=0,4 Mem=4096 GRES_IDX=gpu(IDX:0,2)
>    MinCPUsNode=1 MinMemoryNode=4G MinTmpDiskNode=0
>    Features=(null) DelayBoot=00:00:00
>    OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
>    Command=/dors/csb/home/jsmith/proj/Nav/aMD/5x0m/DMPC/E1784K/eq/./runeq.sb
>    WorkDir=/dors/csb/home/jsmith/proj/Nav/aMD/5x0m/DMPC/E1784K/eq
>    StdErr=/dors/csb/home/jsmith/proj/Nav/aMD/5x0m/DMPC/E1784K/eq/runeq.log
>    StdIn=/dev/null
>    StdOut=/dors/csb/home/jsmith/proj/Nav/aMD/5x0m/DMPC/E1784K/eq/runeq.log
>    Power=
>    GresEnforceBind=Yes
>    TresPerNode=gpu:2

Happy to open a new ticket if it is not related.

Davide
Comment 10 Michael Hinton 2019-03-20 15:13:30 MDT 
Hey Davide,

I'm don't believe they are related, but then again I'm not sure I understand what the problem is.

At any rate, it does help us out if you could file a separate ticket for that.

Thanks!
Michael
Comment 11 Michael Hinton 2019-03-25 14:00:45 MDT 
Marking this as a duplicate. The GPU enforce-binding issue mentioned in comment 9 has been moved to bug 6746. It may likely turn out to be a symptom of the same issue, but until we know for sure, discussion will be continued there.

*** This ticket has been marked as a duplicate of ticket 6370 ***