Hi there,

We have set GrpTRES limits per account (as shown by sacctmgr list assoc tree) in order to avoid a single group to allocate too much resource on the cluster.

I tried to setup a preemptable QoS so that some users can still run preemptable jobs on the whole cluster. By setting a QoS's GrpTRES that matches the whole cluster resources (cpu+gpu), the QoS limit does overrides the assoc GrpTRES (good). However, when preemptable jobs are running and I submit a job using another QoS allowed to preempt, I'm hitting the AssocGrpCpuLimit first.

Indeed the documentation made that clear:

"NOTE: The group limits (GrpJobs, GrpTRES, etc.) are tested when a job is being considered for being allocated resources. If starting a job would cause any of its group limit to be exceeded, that job will not be considered for scheduling even if that job might preempt other jobs which would release sufficient group resources for the pending job to be initiated."
(from http://slurm.schedmd.com/sacctmgr.html)

Still, is there a way to have different per-account TRES limits in different QoSes? That would be very useful in our case.

Thanks!
Stephane Thiell
Stanford Research Computing