Ticket 12047 - How to limit total number of CPUs used by an account, same limit for all accounts?
Summary: How to limit total number of CPUs used by an account, same limit for all acco...
Status: RESOLVED INFOGIVEN
Alias: None
Product: Slurm
Classification: Unclassified
Component: Limits (show other tickets)
Version: 20.02.7
Hardware: Linux Linux
: 3 - Medium Impact
Assignee: Scott Hilton
QA Contact:
URL:
Depends on:
Blocks:
 
Reported: 2021-07-15 14:55 MDT by David Chin
Modified: 2021-07-20 15:55 MDT (History)
1 user (show)

See Also:
Site: Drexel
Slinky Site: ---
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
Google sites: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
NoveTech Sites: ---
Nvidia HWinf-CS Sites: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Tzag Elita Sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed:
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---


Attachments

Note You need to log in before you can comment on or make changes to this ticket.
Description David Chin 2021-07-15 14:55:29 MDT
Hi,

We would like to impose a limit (QOS) on the number of CPUs used in the "def" partition by any account, where all accounts have this same limit.

In our case, we want every account to be able to consume no more than 2,352 CPUs simultaneously in the "def" partition.

I tried defining this:

$ sacctmgr show qos  format=name,priority,grptres,usagefactor
      Name   Priority       GrpTRES UsageFactor
---------- ---------- ------------- -----------
    normal          0                  1.000000
       def          0      cpu=2352    1.000000

And then applying the "def" QOS to the "def" partition, but that seemed to limit the total number of CPUs used on the partition aggregating the usage of all accounts.

How do we accomplish this without applying a QOS to each account separately?

Thanks,
    Dave
Comment 1 Scott Hilton 2021-07-15 16:35:21 MDT
Dave,

I believe you are looking for MaxTRESPerAccount
(see https://slurm.schedmd.com/resource_limits.html)

Run something like this
>sacctmgr update qos def set MaxTRESPerAccount=cpu=2352

Then assign the qos to the partition as you planned.

-Scott
Comment 2 David Chin 2021-07-15 17:55:16 MDT
Hi Scott:

Thanks much.

I've just done that, and I'll do a bit of testing.

--DAve
Comment 3 Scott Hilton 2021-07-19 10:46:10 MDT
David, 

Did that work for you?

-Scott
Comment 4 David Chin 2021-07-19 10:56:06 MDT
Hi, Scott:

Yes, that did.

Thanks very much. Please close this ticket.

Regards,
    Dave
Comment 5 Scott Hilton 2021-07-20 15:55:37 MDT
Closing Ticket