Ticket 4106 - Add accounting records related to the tres usage as calculated by TresBillingWeights and used in fairshare decisions.
Summary: Add accounting records related to the tres usage as calculated by TresBilling...
Status: OPEN
Alias: None
Product: Slurm
Classification: Unclassified
Component: Accounting (show other tickets)
Version: 16.05.7
Hardware: Linux Linux
: 5 - Enhancement
Assignee: Tim Wickberg
QA Contact:
URL:
Depends on:
Blocks:
 
Reported: 2017-08-23 12:14 MDT by Robert Yelle
Modified: 2019-04-09 11:47 MDT (History)
1 user (show)

See Also:
Site: University of Oregon
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
NoveTech Sites: ---
Nvidia HWinf-CS Sites: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Tzag Elita Sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed:
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this ticket.
Description Robert Yelle 2017-08-23 12:14:29 MDT 
Hello,

Is there a way to view TRES and GRES usage used by a job after it completes?  In particular I would like to see the TRESRunMins for a completed job, with contributions from core, memory, and GPU usage.  

Thanks,

Rob
Comment 1 Tim Wickberg 2017-08-23 13:22:13 MDT 
You'd need to add the GRES types to AccountingStorageTRES in slurm.conf, then restart slurmctld.

E.g., 

AccountingStorageTRES=gres/gpu,gres/gpu:k20

CPU, Mem, and Node counts are always captured, but that line will allow you to defined additional TRES usage you'd like to have stored.

After that you should be able to retrieve those values with sacct:

sacct --format=AllocTRES%40

I don't think we have a derived TRES*Min value stored, although you could derive that from AllocTRES * Elapsed.
Comment 2 Tim Wickberg 2017-08-24 16:50:42 MDT 
I'm just flagging this as resolved/infogiven. Please reopen if there's any further questions, or if I can clarify anything.

cheers,
- Tim
Comment 3 Robert Yelle 2017-08-25 13:53:50 MDT 
Hi Tim,

Sorry for the delayed response, I was in meetings all day yesterday and have been in catching up mode all morning.  Anyways, thank you for your response, that is helpful to know.  I do have these related follow-up questions:

Is there a way to display (total) billable TRES?  I see that TRESBillingWeights has an impact on the output of “sshare -al”, but does not have an impact on sreport:

sreport cluster AccountUtilizationByUser start=X end=Y —tres=cpu,mem,gres/gpu format=Accounts,Login,Proper,Used,TresCount

It seems that sreport always displays in terms of raw TRES usage, is that correct?  I know a formula has been provided for computing billable TRES, but would be useful to have independent confirmation of what SLURM is computing as billable TRES from a completed job or the TRES consumed by a user for a particular time period.  Is that currently available?

Also, TresCount is always reported as “0” in sreport for us.  What exactly is TresCount, and why would it report zero if I am tracking TRES?

Thanks,

Rob


On Aug 24, 2017, at 3:50 PM, bugs@schedmd.com<mailto:bugs@schedmd.com> wrote:

Tim Wickberg<mailto:tim@schedmd.com> changed bug 4106<https://bugs.schedmd.com/show_bug.cgi?id=4106>
What    Removed Added
Resolution      ---     INFOGIVEN
Status  UNCONFIRMED     RESOLVED

Comment # 2<https://bugs.schedmd.com/show_bug.cgi?id=4106#c2> on bug 4106<https://bugs.schedmd.com/show_bug.cgi?id=4106> from Tim Wickberg<mailto:tim@schedmd.com>

I'm just flagging this as resolved/infogiven. Please reopen if there's any
further questions, or if I can clarify anything.

cheers,
- Tim

________________________________
You are receiving this mail because:

  *   You reported the bug.
Comment 4 Tim Wickberg 2017-08-25 15:26:49 MDT 
(In reply to Robert Yelle from comment #3)
> Hi Tim,
> 
> Sorry for the delayed response, I was in meetings all day yesterday and have
> been in catching up mode all morning.  Anyways, thank you for your response,
> that is helpful to know.  I do have these related follow-up questions:
> 
> Is there a way to display (total) billable TRES?  I see that
> TRESBillingWeights has an impact on the output of “sshare -al”, but does not
> have an impact on sreport:
> 
> sreport cluster AccountUtilizationByUser start=X end=Y
> —tres=cpu,mem,gres/gpu format=Accounts,Login,Proper,Used,TresCount

TRESBillingWeights only factors into the fairshare calculations at present, there's no version of that calculation stored in the accounting database.

> It seems that sreport always displays in terms of raw TRES usage, is that
> correct?  I know a formula has been provided for computing billable TRES,
> but would be useful to have independent confirmation of what SLURM is
> computing as billable TRES from a completed job or the TRES consumed by a
> user for a particular time period.  Is that currently available?

Not currently. I can open an enhancement request to look into that if you'd like (or convert this bug into one).

> Also, TresCount is always reported as “0” in sreport for us.  What exactly
> is TresCount, and why would it report zero if I am tracking TRES?

This is a bit awkward - after chasing it down through the source, it looks like that field was never implemented, and doesn't print anything right now.
Comment 6 Tim Wickberg 2017-09-27 12:24:27 MDT 
Hi Robert -

I'm reclassifying this as an enhancement request as I'd mentioned previously. Please let me know if you have any further questions.

- Tim
Comment 7 Robert Yelle 2017-09-27 12:54:14 MDT 
Hi Tim,

Thank you for following up, I meant to respond earlier but I was on vacation for a while and am still catching up.

Anyways, yes please make this an enhancement, we would find that very useful.

Thanks,

Rob


On Sep 27, 2017, at 11:24 AM, bugs@schedmd.com<mailto:bugs@schedmd.com> wrote:

Tim Wickberg<mailto:tim@schedmd.com> changed bug 4106<https://bugs.schedmd.com/show_bug.cgi?id=4106>
What    Removed Added
Severity        4 - Minor Issue         5 - Enhancement
Summary way to display TRES and GRES usage      Add accounting records related to the tres usage as calculated by TresBillingWeights and used in fairshare decisions.

Comment # 6<https://bugs.schedmd.com/show_bug.cgi?id=4106#c6> on bug 4106<https://bugs.schedmd.com/show_bug.cgi?id=4106> from Tim Wickberg<mailto:tim@schedmd.com>

Hi Robert -

I'm reclassifying this as an enhancement request as I'd mentioned previously.
Please let me know if you have any further questions.

- Tim

________________________________
You are receiving this mail because:

  *   You reported the bug.