Ticket 3479

I'll note this falls a bit outside our "Level 3" support model, so I'm forced to be a bit terse on these responses. If you have specific questions from the documentation and/or issues on certain functionality, or specific configuration plans you'd like me to review that'd help me know where you're at.

(In reply to Bill from comment #0)
> Folks, 
> Our current need for scheduling jobs on our HPC cluster is pretty simple,
> but a bit more 
> complicated than the default FIFO. Basically, we needs are:
> a) All users are currently considered equal with respect to HPC resource
> utilization
> b) When there is surplus capacity, it should be available without restriction

There's not a good way to do this at the moment; slurm generally wants the same controls and limits in place throughout.

The preferred method to handle that sort of issue is through prioritization of different users/accounts with fairshare.

Fair Tree is our usual suggestion now - https://slurm.schedmd.com/fair_tree.html has an overview of configuring.

You may want to restructure your accounting hierarchy to better account for this.

> c) When there is a backlog (no surplus capacity), we want to assure that any
> given user cannot monopolize more that N% (maybe 60%) of the resources.
> d) For us, resources are generally: CPU, Mem, Nodes

I'd suggest looking into setting limits through GrpTRESMins - out can limit the different resource types (GPUs as well if you add 'AccountingStorageTRES=gpu' to your config) in terms of quantity*time.

You presumably don't want a user loading the empty cluster completely full of 1-month long jobs all at once, but don't care if they were all 10-minutes long. That setting lets you establish a balance between both.

https://slurm.schedmd.com/resource_limits.html

> e) We do not have a large number of users (currently 20 or so), so user
> changes are not too painful unless they get "political".

... limits are all policy though. :)

I'd suggest looking through a few of the older site reports from SLUG (archived here: https://slurm.schedmd.com/publications.html, BYU is probably a good starting point) to get a feel for the different approaches other sites have used in the past.

> We have tried simple resource limits (see previous ticket 3404) but that did
> not work well as it seemed to apply the constraints at the Account level. We
> currently use the Account like department (although that is not necessarily
> how we need to keep it). We have also tried moving the constraints from
> GrpTRES to MaxTRESPU, which did not seem to have any effect in our simple
> testing. 

You may want to add some additional levels. I usually recommend breaking things up (for Universities) into three or four levels, like College -> Department -> Research Group -> (user accounts), to give a bit more flexibility with defining the limits, and a bit more tuning when establishing fairshare.

> Can you recommend a strategy for meeting these needs? Also, if you could
> outline the basic configuration changes/options that would be very helpful. 

I've tried to give a few general pointers, but I will say this can get somewhat complicated, and is one of the things we usually spend a day or two on as part of customer training. Unfortunately, there's not a "one-size-fits-all" approach to modelling the different policies and preferences into fixed configuration; the best pointer I can give is to be willing to test things out and gather data and feedback on different approaches, and try to keep the configuration and setup as simple as you can.

Marking resolved/infogiven; please reopen if there was anything else I could help address.

- Tim