Ticket 8952

Hi Greg, 

The first issue you mentioned looks like it matches the documentation to me. In https://slurm.schedmd.com/sreport.html under cluster utilization the second note says:
"The default view for the "Cluster Utilization" report includes the following fields: Cluster, Allocated, Down, PlannedDown, Idle, Reserved, Reported."

If this isn't what you mean please let me know.

The other two issues look like bugs to me but I will have to do more research. I'll get back to you on them once I know the answer.

-Scott

Regarding the headings - not sure why I got that wrong.

This may help too:

$ sreport   -T gres/gpu,cpu,gres/gpu:v100,gres/gpu:gtx1080ti,gres/gpu:rtx2080ti   cluster Utilization start="2020-04-19T00:00:00" end="2020-04-26T00:00:00"
--------------------------------------------------------------------------------
Cluster Utilization 2020-04-19T00:00:00 - 2020-04-25T23:59:59
Usage reported in TRES Minutes
--------------------------------------------------------------------------------
  Cluster      TRES Name  Allocated       Down PLND Down      Idle   Reserved    Reported 
--------- -------------- ---------- ---------- --------- --------- ---------- ----------- 
   dragon            cpu   93153566   54902197   8669504   7077713   30981166   194784146 
   dragon       gres/gpu    2734732      86964         0   1180063          0     4001760 
   dragon gres/gpu:gtx1+      68318         44         0         0          0       68362 
   dragon gres/gpu:rtx2+      28817      82486         0         0          0      111303 
   dragon  gres/gpu:v100    1006816       3104         0         0          0     1009920

Hi Greg, 

Thanks for the other report that helped. To fully identify the issue could you share a database dump. I'm not sure if the issue is in the database values or in the sreport function.

For reference this is how you make a database dump:
mysqldump --databases <your_database> > <file>

Please zip the file as well:
gzip <file>

-Scott

Have taken dump of db. Compressed it's 1.1GB

Can it be uploaded to somewhere? (Or do you want it attached to this ticket?)

  -greg

Greg,

That is bigger than the maximum file size for this site. So it won't work here.

You could share it via google drive (or a similar service) with scott@schedmd.com.

-Scott

Greg,

How do you know that 74% down is not correct?

Could you send me your slurm.conf and any included .conf files. I want to see how the nodes are set up to help me identify patterns in what is reported as down.

Also, I got your database working and have reproduced same issues on my machine now.

I'm still not sure what is causing reported to be greater than 100% but I am looking into it.

Thanks, Scott

Greg,

Also send your gres.conf if you can.

Thanks,

Scott

(In reply to Scott Hilton from comment #7)
> Greg,
> 
> How do you know that 74% down is not correct?
> 
> Could you send me your slurm.conf and any included .conf files. I want to
> see how the nodes are set up to help me identify patterns in what is
> reported as down.
> 
> Also, I got your database working and have reproduced same issues on my
> machine now.
> 
> I'm still not sure what is causing reported to be greater than 100% but I am
> looking into it.
> 
> Thanks, Scott

I think I forgot to enable notifications on this comment.

Can you send me your slurm.conf and included files?

Thanks, Scott

Greg, 

Part of the fix for bug 8958 I believe will also fix the issue where you were getting percentages much larger than %100. This is the commit:

https://github.com/SchedMD/slurm/commit/010d752b275a1ccb1c9537238233c938f2412ec2

Could you explain how you know that 74% down for the rtx is not correct? 
Also, do you have any other info about the rtx setup that may be relevant.

Take care,

Scott

Hi Scott,

The evidence that the 2080s were up is stored in our slurm database (we use slurm to report when nodes are down etc).

We have 32 x RTX2080ti in 4 nodes (8 GPUs per node).

During the week in question one node was down (all week) due to a GPU fault.

The remaining three nodes were processing jobs (all week).

For slurm to report that the 2080Ti was down 74.11% would have meant that three of the four nodes were not available for virtually the entire week (just shy of 75% down).

We are confident this was not the case.

(BTW - I sent all out configs as a dropbox link - did you get it?)

   -greg

Greg, 

Yes, I did get your configs. They were quite helpful in diagnosing the problems for this bug and bug 8958.

Thanks for the info. I'll keep you updated on that last issue.

-Scott

Greg,

It looks like the total time for the one down graphics card is correct. The percent was off because we had an incorrect tres count 8 instead of 24. This should also be fixed be the change in 8958. 

If you had had the fix from 8958, I would guess you would get something like:
>Cluster      TRES Name   Allocated       Down  PLND Down      Idle   Reserved    Reported 
>dragon  gres/gpu:rtx2+         ~9%       ~26%      0.00%      ~65%      0.00%        100%    

That means though that we still have only 9% of the time was allocated. This could mean that the rest of the time the gpus were actually idle or there is a bug counting the allocated time (which I am looking into). 

If it was idle time  this could result from nodes running just on the cpus or jobs reserving the nodes until they have all the resources to run.

Have you encountered any similar bugs to this? 

Also, have you updated to 19.05.7 or 20.02.3?

-Scott

Hi Scott,

A ran a simple script over the output of sacct -X for the week in question. The results are:

gpu504-37 0 GPU seconds (0% of period)
gpu510-02 4224835 GPU seconds (87% of period)
gpu510-12 3199137 GPU seconds (66% of period)
gpu510-17 2431046 GPU seconds (50% of period)

(GPU seconds is the number of seconds during the week that the GPU was allocated to a job. The percentage is calculated between GPU seconds available [of 8 GPUs] and the overall number of seconds in the week).

Our slurm version is currently:

$ scontrol -V
slurm 19.05.5

We plan on upgrading to 20 in the next maintenance session (in three weeks).
In the interim we can upgrade to a later version of 19 if it is recommend.

Greg, 

Could you send me that script?

Thanks,

Scott

Hi Scott,

Script is at https://gitlab.com/greg.wickham/python-slurm

When the script run it identifies a job that spans more than one node. Looking at this job further (# 10342997) I'm not sure how to update the script to keep track of the correct allocations.

Maybe these following questions should be a separate ticket, but they are related to the data for this ticket. (Let me know if I should open a separate ticket)

For job 10342997

$ sacct -P -j 10342997 --format jobidraw,start,end,nodelist,reqtres,reqgres,alloctres,allocgres
JobIDRaw|Start|End|NodeList|ReqTRES|ReqGRES|AllocTRES|AllocGRES
10342997|2020-04-21T18:33:34|2020-04-21T18:33:50|gpu504-37,gpu510-17|billing=20,cpu=20,gres/gpu=4,mem=120G,node=1|gpu:8|billing=20,cpu=20,gres/gpu=8,mem=120G,node=2|gpu:8
10342997.batch|2020-04-21T18:33:34|2020-04-21T18:33:50|gpu504-37||gpu:8|cpu=10,mem=60G,node=1|gpu:8
10342997.extern|2020-04-21T18:33:34|2020-04-21T18:33:51|gpu504-37,gpu510-17||gpu:8|billing=20,cpu=20,gres/gpu=8,mem=120G,node=2|gpu:8
10342997.0|2020-04-21T18:33:36|2020-04-21T18:33:52|gpu510-17||gpu:8|cpu=1,gres/gpu:gtx1080ti=4,gres/gpu=4,mem=6G,node=1|gpu:8

$ sinfo -n gpu510-17,gpu504-37 -o "%n,%G"
HOSTNAMES,GRES
gpu504-37,gpu:gtx1080ti:8(S:0-1)
gpu510-17,gpu:rtx2080ti:8(S:0-1)

Q1: The job "10342997.0" indicates using "gres/gpu:gtx1080ti=4" on node gpu510-17, but the node has "rtx2080ti"

Q2: 10342997.0 shows AllocTRES is 4 GPUs while AllocGRES is 8 GPUS

Q3: 10342997.batch and 10342997.0 are running at the same time on different nodes - what is the total of GPUs allocated?

Greg,

Could you post those questions in a new ticket? We will just focus on the original question involving missing allocation time in this thread. 

This will help us keep track of the issues.

Thanks, 

Scott

created https://bugs.schedmd.com/show_bug.cgi?id=9159

Greg,

The numbers from your script look like they are in the right ballpark to me. This tells me that the direct accounting records are probably correct. The issue is probably in adding up the time. 

I'm not sure but I think there is a good chance that the previous fix (see 8958) could also fix this discrepancy where we are missing allocation time, Leading to 74% down. 

Once you update to 20.02.3 or 19.05.7, if you see any new issues like this let me know. 

I'm going to close this ticket for now, but if the same issue appears again please reopen it so we can address the bug.

Good luck,

Scott