Ticket 10118

Summary:	Slurm Fairschare Configuration
Product:	Slurm	Reporter:	Marcus Boden <mboden>
Component:	Configuration	Assignee:	Scott Hilton <scott>
Status:	RESOLVED INFOGIVEN	QA Contact:
Severity:	4 - Minor Issue
Priority:	---	CC:	ben
Version:	20.02.5
Hardware:	Linux
OS:	Linux
Site:	GWDG	Alineos Sites:	---
Atos/Eviden Sites:	---	Confidential Site:	---
Coreweave sites:	---	Cray Sites:	---
DS9 clusters:	---	HPCnow Sites:	---
HPE Sites:	---	IBM Sites:	---
NOAA SIte:	---	OCF Sites:	---
Recursion Pharma Sites:	---	SFW Sites:	---
SNIC sites:	---	Linux Distro:	---
Machine Name:		CLE Version:
Version Fixed:		Target Release:	---
DevPrio:	---	Emory-Cloud Sites:	---

Description Marcus Boden 2020-11-03 08:13:29 MST

Dear SchedMD support,

we have trouble getting the fairshare quite the way we want it. Our situation:
1. Most of the nodes we host are our own and are shared among all users.
2. Some departments buy their own nodes which they then exclusively use (they're still allowed to use our nodes)
3. Some departments buy nodes to be integrated among our nodes for everyone in exchange of a higher fairshare.

The first two cases are fine, a problem comes in the third category.
We currently have an "all" account, where all the users are in. Most users are just normal users of category 1 and they get 1000 shares.
Users of category 2 get 1000 shares in all, but also an extra account to use on their own nodes (in their own partition).
For the third category, we calculate how much their additional nodes add to our cluster and add that to their shares, e.g. if the department's nodes add 5% to our cluster, they will get 1050 shares.

The problem with this setup is that the more people work in a department, the more users get higher fairshare, leading to an advantage for bigger departments. What we actually want is a setup where users of the third category share a pool of extra shares and once this pool is depleted, they use their default 1000 shares that everybody has.
Is something like that possible with fairshare in Slurm?

Best,
Marcus

Comment 3 Scott Hilton 2020-11-03 17:07:29 MST

Marcus,

There are two fairshare algorithms: The fair tree algorithm and the classic. The classic algorithm is only active if you have PriorityFlags=NO_FAIR_TREE in you slurm.conf. I will assume you are using the current default algorithm which is fair tree.

For the fair tree, fair share is calculated recursively on each account level. Shares only matter in the ratio between sibling accounts. (https://slurm.schedmd.com/fair_tree.html)

For the department to have 5% of the shares you could do something like this:
>sacctmgr show assoc tree format=cluster,account,user,share,parentname
>   Cluster              Account       User     Share   Par Name 
>---------- -------------------- ---------- --------- ----------
>    dragon root                                    1 
>    dragon  all                                   95       root
>    dragon   all                     scott      1000
>    dragon   all                    marcus      1000
>    dragon  department                             5       root
>    dragon   department              scott         1
Since 'all' and 'department' are on the same level their shares interact. Since the children of 'all' each have 1000 shares %50 of whatever is allotted to 'all'.

User scott would have two accounts to think about. He would have to decide which account to use for any given job. Sadly, there isn't a way to automatically draw from the ideal account. A user could run sshare to see which account has more fairshare at any time before running a job.

If you don't want your users to have to think about which accout to use based on fairshare, you could remove the 'all' account and split it among departments directly:
>sacctmgr show assoc tree format=cluster,account,user,share,parentname
>   Cluster              Account       User     Share   Par Name 
>---------- -------------------- ---------- --------- ----------
>    dragon root                                    1 
>    dragon  math                                 100       root
>    dragon   math                      tom         2
>    dragon   math                      bob         1
>    dragon  engineering                          110       root
>    dragon   engineering             scott      9534
>    dragon  physics                              100       root
>    dragon   physics                 marcus        1
In this example, Engineering might deserve more and so gets 110 shares out of 310. Tom and Bob each use the the math department but Tom might get more shares in that department. Scott and Marcus each have %100 in their department.

Do you have any questions or concerns? Anything I should explain more?

-Scott

Comment 4 Scott Hilton 2020-11-12 12:58:54 MST

If you have any follow up questions feel free to reopen this ticket.

-Scott