Ticket 15737 - Root user start job from queue bypassing accounts and queue limits
Summary: Root user start job from queue bypassing accounts and queue limits
Status: RESOLVED TIMEDOUT
Alias: None
Product: Slurm
Classification: Unclassified
Component: Limits (show other tickets)
Version: - Unsupported Older Versions
Hardware: Linux Linux
: 4 - Minor Issue
Assignee: Director of Support
QA Contact:
URL:
Depends on:
Blocks:
 
Reported: 2023-01-05 08:40 MST by Jay McGlothlin
Modified: 2023-01-27 10:29 MST (History)
0 users

See Also:
Site: RPI/CCNI - Rensselaer Polytechnic Institute
Slinky Site: ---
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
Google sites: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
NoveTech Sites: ---
Nvidia HWinf-CS Sites: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Tzag Elita Sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed:
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this ticket.
Description Jay McGlothlin 2023-01-05 08:40:19 MST 
Is there a recommended way for a root user to move a job in the queue from Pending reason AssocMaxJobsLimit to Running manually?

The use case is that we set MaxJobs on accounts to limit users from single handedly filling the cluster with small jobs in parallel.  However, every once in a while this will result in a lot of idle nodes if the queue is empty except for that user.  I am looking for an easy way for our admins to release another group of pending jobs to run when they see this.
Comment 1 Caden Ellis 2023-01-09 16:18:31 MST 
Jay,

Specifically for what you want, you need to override the limit at the user level. Once the limit is changed, the scheduler will evaluate the jobs again but check the new limits and allow more jobs to run. This will take a several seconds to happen once the limit is changed.

Example:

sacctmgr modify user where name=<username> set maxjobs=<more jobs>

Then once more are scheduled, change it back.

sacctmgr modify user where name=<username> set maxjobs=<old amount>

There may be other limits that you could mix and match instead to avoid needing to manually override like this while still preventing total cluster usage:

https://slurm.schedmd.com/resource_limits.html

But this manual override I suggested will allow more jobs to run for that user until you change it back, like you requested. When I raised the limit and saw in squeue that more got scheduled, I immediately lowered the limit again (so I didn't have to wait for all of the jobs to finish to lower that limit). I saw no issues handling it like this. 

Does this answer your question?

Caden
Comment 2 Caden Ellis 2023-01-17 14:45:23 MST 
Do you have an update for me on this?
Comment 3 Caden Ellis 2023-01-27 10:29:09 MST 
Feel free to open this back up if you have further questions.

Caden