Ticket 15341

Summary:	Managing CPU and GPU allocations within a single SLURM cluster
Product:	Slurm	Reporter:	Maciej Cytowski <maciej.cytowski>
Component:	Accounting	Assignee:	Ben Roberts <ben>
Status:	RESOLVED INFOGIVEN	QA Contact:
Severity:	4 - Minor Issue
Priority:	---
Version:	22.05.2
Hardware:	Linux
OS:	Linux
Site:	Pawsey	Alineos Sites:	---
Atos/Eviden Sites:	---	Confidential Site:	---
Coreweave sites:	---	Cray Sites:	---
DS9 clusters:	---	HPCnow Sites:	---
HPE Sites:	---	IBM Sites:	---
NOAA SIte:	---	OCF Sites:	---
Recursion Pharma Sites:	---	SFW Sites:	---
SNIC sites:	---	Linux Distro:	---
Machine Name:		CLE Version:
Version Fixed:		Target Release:	---
DevPrio:	---	Emory-Cloud Sites:	---

Description Maciej Cytowski 2022-11-03 02:27:08 MDT

Hi,

allocations and access for projects running on our current CPU-only system are managed through SLURM accounts and shares. On the new heterogeneous system we would like to manage CPU and GPU allocations separately i.e. each project (account) will have a separate CPU and GPU allocation with separate billing. 

One obvious solution is to create separate slurm accounts for each project (project-cpu and project-gpu for each project), however we would like to avoid that.

Are there any examples or best practices on how this can be implemented? 

Kind regards,
Maciej

Comment 1 Ben Roberts 2022-11-04 10:37:40 MDT

Hi Maciej,

Creating unique accounts for CPU and GPU type jobs is the first thing that I would recommend for keeping the billing separate.  It would allow you to easily see the usage of the two types of jobs.  I can understand the desire to avoid creating two versions of each Account you currently have in place though.  

One option that may work for you would be to use Workload Characterization Keys (WCKeys) to identify the different types of jobs.  This would be an extra flag that is added to the different types of jobs so that you can easily identify CPU vs GPU jobs when they use the same Accounts.  

You can read more about WCKeys in the documentation here:
https://slurm.schedmd.com/wckey.html

It's also worth noting that sreport does have reports that take WCKeys into account:
https://slurm.schedmd.com/sreport.html#SECTION_REPORT-TYPES

Let me know if this sounds like something that would work for you.

Thanks,
Ben

Comment 2 Ben Roberts 2022-11-30 12:07:54 MST

Hi Maciej,

Did either of the proposed solutions sound like they would work for you?  Let me know if you still need help with this ticket or if it's ok to close.

Thanks,
Ben

Comment 3 Maciej Cytowski 2022-12-13 21:39:08 MST

Hi Ben, 

thank you for your help. This can be closed now.

Kind regards,
Maciej

Comment 4 Ben Roberts 2022-12-14 08:49:41 MST

I'm glad you found a solution that will work for you.  Let us know if there's anything we can do to help in the future.

Thanks,
Ben