Ticket 10749

Summary: Add sreport option to report cluster user utilization by partition
Product: Slurm Reporter: Alan Sill <Alan.Sill>
Component: AccountingAssignee: Ben Roberts <ben>
Status: RESOLVED INFOGIVEN QA Contact:
Severity: 4 - Minor Issue    
Priority: --- CC: misha.ahmadian
Version: 20.11.3   
Hardware: Linux   
OS: Linux   
Site: TTU Alineos Sites: ---
Atos/Eviden Sites: --- Confidential Site: ---
Coreweave sites: --- Cray Sites: ---
DS9 clusters: --- HPCnow Sites: ---
HPE Sites: --- IBM Sites: ---
NOAA SIte: --- OCF Sites: ---
Recursion Pharma Sites: --- SFW Sites: ---
SNIC sites: --- Linux Distro: ---
Machine Name: CLE Version:
Version Fixed: Target Release: ---
DevPrio: --- Emory-Cloud Sites: ---

Description Alan Sill 2021-01-30 12:24:11 MST
sreport does not seem to have options either to limit total user utilization reports by partition or (better) to include the partition utilization as columns in the cluster reports. Can this be added, please? If there is a way to do this, it would be apprecaited if you could point it out in the documentation.
Comment 1 Ben Roberts 2021-02-01 09:38:54 MST
Hi Alan,

You're right that there isn't currently a way to have the partition information included in a user report with sreport.  It should be possible to do this, but would require sponsorship for the development of this additional feature.  If you're interested in sponsoring this work let me know and we can look into it further. 

As an alternative, you may be able to get the information you need by using WCKeys to tag different types of jobs.  You can see that there are various places where you can use the WCKey in sreport, but the one that would probably be most useful to you would be the Cluster UserUtilizationByWckey report.  

The way I would recommend adding a WCKey to each of the jobs would be to create WCKeys that correspond to the different partitions you use and then have a submit filter and adds the appropriate WCKey to each job based on the requested partition.  You can read more about WCKeys here:
https://slurm.schedmd.com/wckey.html

Let me know if you have any questions about this.

Thanks,
Ben
Comment 2 Ben Roberts 2021-02-23 08:14:06 MST
Hi Alan,

Do you have any additional questions about using WCKeys to get the information you want from sreport?

Thanks,
Ben
Comment 3 Alan Sill 2021-02-27 08:26:28 MST
Ideally, we would be able to query the information directly from the database by partition. It seems odd to me that this information is not considered a basic part of the report. Adding WCKeys just to get it does not seem to me to be an optimal solution.
Comment 4 Ben Roberts 2021-03-01 08:53:10 MST
If you're looking for a way to get this information outside of sreport then you should be able to do this with sacct.  There is information recorded about jobs that is stored in the database that you can then query with sacct.  The sacct command allows you to set various constraints on the start and end time, along with various attributes of the jobs like the partition, account, user, node(s), etc.  It will return a list with information about the jobs that have the specified attributes.  There are also quite a few attributes you can choose to display about the jobs that are returned.  For what you are describing I would recommend something like this:

sacct --start=2021-02-01 --end=2021-02-28 --user=user1 -o "jobid,user,partition,allocnodes,alloccpus,ntasks"

I just added a few example fields for the output specifier, you can obviously customize this to display what you are interested in.  For a comprehensive list of the options available and fields available to print you can look at the man page for sacct or the same information on our website:
https://slurm.schedmd.com/sacct.html

This doesn't put the information together nicely for you, but should allow you to get the information you want to report on.  Let me know if this seems like a better solution for you or still isn't quite what you're looking for.

Thanks,
Ben
Comment 5 Ben Roberts 2021-03-26 08:31:56 MDT
Hi Alan,

I wanted to see if the suggestion in my last comment sounds like it will work.  Let me know if you have any additional questions about this or if this ticket is ok to close.

Thanks,
Ben
Comment 6 Alan Sill 2021-03-26 08:32:08 MDT
I will be out of the office Monday March 15 through Monday March 29, 2021, inclusive. For HPCC management issues, please contact Dr. Thomas Brown (thomas.brown@ttu.edu. For HPCC operational issues, please contact Ms. Amy Wang (amy.wang@ttu.edu) or Mr. Sowmith Lakki Reddy (sowmith.lakki-reddy@ttu.edu). For HPCC business topics please contact Ms. Amanda McConnell (amanda.mcconnell@ttu.edu) or call 742-4350. Send email to hpccsupport@ttu.edu for HPCC cluster or user support topics.
Comment 7 Ben Roberts 2021-04-15 15:56:21 MDT
Hi Alan,

I haven't heard any follow up questions and I believe the information I sent should have helped.  I'll close this ticket, but feel free to let me know if something else does come up.

Thanks,
Ben