Ticket 8024

Hi Greg,

Indeed, AllocTRES in `scontrol show nodes` can be messed up when there is more than one GPU type. Differing `Cores=` definitions for each of the GPUs in gres.conf can also increase the chances of this bug showing up. Although AllocTRES in `scontrol show nodes` is sometimes incorrect, I haven't seen any evidence yet that jobs get scheduled with incorrect resources, or that AllocTRES in sacct in wrong.

I actually have a fix pending review in a private internal bug for this very issue (bug 7827). I'll let you know when that lands.

In the meantime, could you give me your current slurm.conf and gres.conf, so I can reproduce your issue and make sure my patch solves your issue? And I'm assuming that sacct is showing the correct GRES request for the job, but if not, I'll need that as well.

Thanks,
-Michael

Created attachment 12182 [details]
slurm.conf

Created attachment 12183 [details]
gres.conf

Hi Michael,

We're not seeing what you describe - the output of 'scontrol show nodes' is correct.

The user requested 'v100' GPUs and a suitable node was chosen (gpu208-02).

The job steps ran on the selected node (with v100 GPUs which is correct).

The issue is the "step" accounting is showing incorrect AllocTRES - the node has v100 GPUs but the step AllocTRES is indicating 'gtx1080ti'.

FWIW, the user started with an 'salloc' and then used 'srun' to run steps within the allocation.

Files uploaded.

   -Greg

(In reply to Greg Wickham from comment #5)
> We're not seeing what you describe - the output of 'scontrol show nodes' is
> correct.
Ok, thanks. I'll get to the bottom of this and get back to you.

-Michael

Sorry Greg, could you also give me the node definition for gpu208-02? I didn't realize it would be in a different file. I want to double check that there isn't anything fishy going on there. Thanks.

Also, if you know the commands used to reproduce the jobs with the issue, that would be helpful. Thanks.

And if you see any currently-running jobs with this problem, an `scontrol show job <job_id>` would be nice as well.

NodeName=gpu208-[02,06,10,14,18] Gres=gpu:v100:8 Feature=dragon,ibex2018,nolmem,cpu_intel_platinum_8260,intel,gpu,intel_gpu,local_200G,local_400G,local_500G,gpu_v100,v100,nossh RealMemory=763000 Boards=1 SocketsPerBoard=2 CoresPerSocket=24 ThreadsPerCore=1 Weight=32000


I don't know the commands used. I do have a belief that the user did an "salloc" to obtain the resources and then used "srun" to use them.

I'll have a look tomorrow and see if I can find any active jobs exhibiting similar behaviour.

Good news: I was able to reproduce the bug!

*****************
Reproducer
*****************

slurm.conf:
************************************
NodeName=DEFAULT CPUs=48 Sockets=2 CoresPerSocket=24 ThreadsPerCore=1 State=UNKNOWN RealMemory=7000
GresTypes=gpu
DebugFlags=gres
SlurmdParameters=config_overrides
AccountingStorageTRES=gres/gpu,gres/gpu:gtx1080ti,gres/gpu:p100,gres/gpu:p6000,gres/gpu:rtx2080ti,gres/gpu:tesla_k40m,gres/gpu:v100,cpu,node
NodeName=test1 NodeAddr=localhost Port=19052 Gres=gpu:v100:8


gres.conf:
************************************
Name=gpu Type=v100 File=/dev/tty[0-7]


Commands:
************************************
hintron@hintron:~/slurm/19.05/extra$ salloc --gres=gpu:v100:4
salloc: Granted job allocation 119

hintron@hintron:~/slurm/19.05/extra$ srun --gres=gpu:4 sleep 60 &                      
[1] 18663

hintron@hintron:~/slurm/19.05/extra$ sacct --format jobid,nodelist,alloctres%60 -j 119
      JobID        NodeList                                                    AllocTRES  
------------ ---------------        -----------------------------------------------------  
119                    test1   billing=1,cpu=1,gres/gpu:v100=4,gres/gpu=4,mem=150M,node=1  
119.0                  test1           cpu=1,gres/gpu:gtx1080ti=4,gres/gpu=4,mem=0,node=1  

hintron@hintron:~/slurm/19.05/extra$ scontrol show nodes  
NodeName=test1 Arch=x86_64 CoresPerSocket=24  
  CPUAlloc=1 CPUTot=48 CPULoad=0.65
  AvailableFeatures=(null)
  ActiveFeatures=(null)
  Gres=gpu:v100:8
  NodeAddr=localhost NodeHostName=test1 Port=19052 Version=19.05.3-2
  OS=Linux 5.3.0-19-generic #20-Ubuntu SMP Fri Oct 18 09:04:39 UTC 2019  
  RealMemory=7000 AllocMem=150 FreeMem=539 Sockets=2 Boards=1
  State=MIXED ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A
  Partitions=debug  
  BootTime=2019-11-04T11:06:33 SlurmdStartTime=2019-11-04T12:49:55
  CfgTRES=cpu=48,mem=7000M,billing=48,gres/gpu=8,gres/gpu:v100=8
  AllocTRES=cpu=1,mem=150M,gres/gpu=4,gres/gpu:v100=4
  CapWatts=n/a
  CurrentWatts=0 AveWatts=0
  ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s
 


However, specifying gpu:v100:4 explicitly for both salloc and srun makes it work as expected:
************************************

hintron@hintron:~/slurm/19.05/extra$ salloc --gres=gpu:v100:4
salloc: Granted job allocation 118

hintron@hintron:~/slurm/19.05/extra$ srun --gres=gpu:v100:4 sleep 60 &                      
[1] 18496

hintron@hintron:~/slurm/19.05/extra$ sacct --format jobid,nodelist,alloctres%60 -j 118
      JobID        NodeList                                                    AllocTRES  
------------ ---------------        -----------------------------------------------------  
118                    test1   billing=1,cpu=1,gres/gpu:v100=4,gres/gpu=4,mem=150M,node=1  
118.0                  test1                cpu=1,gres/gpu:v100=4,gres/gpu=4,mem=0,node=1  

hintron@hintron:~/slurm/19.05/extra$ scontrol show nodes                               
NodeName=test1 Arch=x86_64 CoresPerSocket=24  
  CPUAlloc=1 CPUTot=48 CPULoad=0.24
  AvailableFeatures=(null)
  ActiveFeatures=(null)
  Gres=gpu:v100:8
  NodeAddr=localhost NodeHostName=test1 Port=19052 Version=19.05.3-2
  OS=Linux 5.3.0-19-generic #20-Ubuntu SMP Fri Oct 18 09:04:39 UTC 2019  
  RealMemory=7000 AllocMem=150 FreeMem=586 Sockets=2 Boards=1
  State=MIXED ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A
  Partitions=debug  
  BootTime=2019-11-04T11:06:33 SlurmdStartTime=2019-11-04T12:49:55
  CfgTRES=cpu=48,mem=7000M,billing=48,gres/gpu=8,gres/gpu:v100=8
  AllocTRES=cpu=1,mem=150M,gres/gpu=4,gres/gpu:v100=4
  CapWatts=n/a
  CurrentWatts=0 AveWatts=0
  ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s


I did a few more tests, and here’s a summary of the GRES combinations for salloc and srun:
**********************************
salloc      | srun       | sacct
---------------------------------
gpu:4       | gpu:4      | gres/gpu:gtx1080ti=4
gpu:4       | gpu:v100:4 | Invalid GRES specification
gpu:v100:4  | gpu:4      | gres/gpu:gtx1080ti=4
gpu:v100:4  | gpu:v100:4 | gres/gpu:v100=4

So it looks like the current work-around is to instruct users to explicitly specify the GPU type (v100) for both salloc AND srun, where possible. 

I’ll investigate this some more and hopefully get back to you once I have a fix.

Thanks,
-Michael


P.S. In gres.conf, `Count` is completely redundant for GPUs, since the count is inferred from the `File` is specification.

Hey Greg,

It looks like this is a known issue. From https://slurm.schedmd.com/slurm.conf.html#OPT_AccountingStorageTRES:

"If a job requests GPUs, but does not explicitly specify the GPU type, then its resource allocation will be accounted for as either "gres/gpu:tesla" or "gres/gpu:volta", although the accounting may not match the actual GPU type allocated to the job and the GPUs allocated to the job could be heterogeneous. In an environment containing various GPU types, use of a job_submit plugin may be desired in order to force jobs to explicitly specify some GPU type."

I still don't think this is ideal and hope to improve on this in 20.02 or later. But it looks like for now, using the job_submit plugin to alter any user's "srun --gres:gpu:4" to "srun --gres:gpu:v100:4" on nodes that only have a single type is your best bet to avoid this issue.

Another thing you can do that might help mitigate the issue is to add an additional 'meta' TRES GPU storage type to AccountingStorageTRES, like `gres/gpu:a_typeless`. Whatever it is named, it just needs to be higher in the alphabet than `gtx1080ti` (`a` comes before `g`). This is because when `srun --gres=gpu:4` is specified inside the allocation, since type is null, Slurm currently just picks the first GPU type it finds in a sorted list taken from AccountingStorageTRES.

It's clunky, but at least you will know what's going on and that the step didn't specify a type (so it could be any of the GPUs within the allocation).

For example:

$ sacct --format jobid,nodelist,alloctres%60 -j 140
       JobID        NodeList                                                    AllocTRES 
------------ ---------------        ----------------------------------------------------- 
140                    test1   billing=1,cpu=1,gres/gpu:v100=4,gres/gpu=4,mem=150M,node=1 
140.0                  test1          cpu=1,gres/gpu:a_typeless=4,gres/gpu=4,mem=0,node=1 

Now this is less confusing, because Slurm doesn't tell us that we have gtx1080ti GPUs. In this case, we can infer that the 4 GPUs were v100s, since it's confined to the job's allocation of v100s.

For allocations with multiple GPU types, this could still mean that you won't know exactly which GPU types were on the step, but it at least won't be misleading.

Thanks,
-Michael

Hi Michael,

Thanks for the comprehensive response.

We currently have 5 different GPU types in use, so I doubt there is an "easy" way to fix this during job submission (the target host isn't available during job submission so there's no way to generalize the GPU type).

We'll just have to deal with the step GPU TRES (probably) being incorrect.

Please close the ticket (unless you have something more to add).

cheers,

   -Greg

Ok Greg, thanks. I'll keep you posted if there are any new developments in this area.

-Michael

*** Ticket 9275 has been marked as a duplicate of this ticket. ***

*** Ticket 9932 has been marked as a duplicate of this ticket. ***

*** Ticket 9543 has been marked as a duplicate of this ticket. ***

Dear Michael,
Any estimation about when this issue will be solved?
We've upgraded recently to 20.11 and the issue is still there.
Thanks,
Ahmed

Hi Ahmed,

We are looking at a comprehensive fix to this problem for 21.08, but can't guarantee a timeline right now.

-Michael

Hello Greg and Ahmed,

This issue should now be resolved in the upcoming 21.08 release in the following commits: https://github.com/SchedMD/slurm/compare/8aab06e4a120...0ff1668043e3.

Thanks!
-Michael