SchedMD - Slurm Support – Full Text Ticket Listing
- Home
- | New
- | Browse
- | Search
- | [?]
- | Reports
- | Help
- | New Account
- | Log In
- | Forgot Password
|
Description
Kilian Cavalotti
2017-05-30 13:17:00 MDT
Hello Killian, I tried to go to https://stanford.box.com/v/slurmconf to see the whole slurm.conf but it requires a login and password. Can you just attach it to the bug? Thanks Tim Hi Tim, My bad. It should be better now, if you want to take another shot at it. Thanks, -- Kilian Created attachment 4663 [details]
Logging patch
I put together a logging patch that logs a few more things at the info level to help make it apparent why the controller thinks your user is over their limit. Please apply the logging patch and have the user reproduce the error. Then paste in the new log message. This is the log line that I added a few new fields to:
job submit for user csfoo(36268): qos max submit job limit exceeded 3000, current jobs <>, newly submitted job count <>
The first number is the limit (3000). The second number is the current number of jobs (pending or running) and the third is the number of jobs in the job submission (usually 1 unless it's an array job).
Also if you could paste in the output of:
squeue -u csfoo
That would be helpful.
Thanks
Tim
Hi Tim, (In reply to Tim Shaw from comment #4) > I put together a logging patch that logs a few more things at the info level > to help make it apparent why the controller thinks your user is over their > limit. Please apply the logging patch and have the user reproduce the > error. Thanks, I'll work on that. And provide the squeue output too. Cheers, -- Kilian Hi Tim, Sorry for the delay, the issue only manifests itself after a bunch of jobs have been submitted, typically 12-24 hours. Here is what the patched slurmctld logged when this happened again: -- 8< --------------------------------- Jun 2 04:19:43 sh-sl01 slurmctld[85048]: job submit for user csfoo(36268): qos max submit job limit exceeded 3000, current jobs 3000, newly submitted job count 1 Jun 2 04:19:43 sh-sl01 slurmctld[85048]: _job_create: exceeded association/qos's limit for user 36268 Jun 2 04:19:43 sh-sl01 slurmctld[85048]: _slurm_rpc_submit_batch_job: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits) -- 8< --------------------------------- So it looks like the scheduler thinks there are 3000 jobs from that user in queue at that time, but there hasn't been. I'm attaching both the squeue output for that user (squeue -u csfoo) at the time this happened, and a graph of the total number of pending and running jobs for the whole cluster. Here's an excerpt of the job submission tries from that user (t8.txt is the attached squeue output): -- 8< --------------------------------- 2017-06-02 04:19:55 [csfoo@sh-ln02]$ /usr/bin/srun -J sdev -t 01:00:00 -p akundaje --mem 4GB --pty /bin/bash -i -l srun: error: Unable to allocate resources: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits) 2017-06-02 04:19:58 [csfoo@sh-ln02]$ /usr/bin/srun -J sdev -t 01:00:00 -p akundaje --mem 4GB --pty /bin/bash -i -l srun: error: Unable to allocate resources: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits) 2017-06-02 04:19:59 [csfoo@sh-ln02]$ /usr/bin/srun -J sdev -t 01:00:00 -p akundaje --mem 4GB --pty /bin/bash -i -l srun: error: Unable to allocate resources: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits) 2017-06-02 04:20:00 [csfoo@sh-ln02]$ squeue -u csfoo --Format=jobid,partition,name,state,timeused,timelimit,numnodes,numcpus,nodelist > t8.txt -- 8< --------------------------------- Then apparently, when he tried to submit a job to multiple partitions, it "unblocked" the situation: -- 8< --------------------------------- 2017-06-02 04:25:55 [csfoo@sh-ln02]$ /usr/bin/srun -J sdev -t 01:00:00 -p akundaje --mem 4GB --pty /bin/bash -i -l srun: error: Unable to allocate resources: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits) 2017-06-02 04:25:57 [csfoo@sh-ln02]$ /usr/bin/srun -J sdev -t 01:00:00 -p owners,akundaje --mem 4GB --pty /bin/bash -i -l srun: job 170981 queued and waiting for resources srun: job 170981 has been allocated resources [csfoo@sh-102-31]$ exit logout 2017-06-02 04:26:13 [csfoo@sh-ln02]$ /usr/bin/srun -J sdev -t 01:00:00 -p akundaje --mem 4GB --pty /bin/bash -i -l [csfoo@sh-102-31]$ exit logout 2017-06-02 04:26:19 [csfoo@sh-ln02]$ -- 8< --------------------------------- I'm also attaching the full slurmctld.log for that time period. Thanks! -- Kilian Created attachment 4690 [details]
slurmctld.log
Created attachment 4691 [details]
Number of jobs
That graph shows the total number of pending and running jobs on the whole cluster over the last 24 hours. It never reached 3,000 globally, let alone for a single user.
Created attachment 4692 [details]
squeue output
Killian, I'm still investigating this problem but I wanted to know if you see any different squeue results with the users's jobs if you run: squeue -u csfoo -t pd,r,s,cg,cd,cf,ca,f,to,pr,bf,nf,rv,SPECIAL_EXIT I just want to make sure that the jobs aren't in a different state but still somehow being counted against the users's limit. Thanks Tim (In reply to Tim Shaw from comment #12) > I'm still investigating this problem but I wanted to know if you see any > different squeue results with the users's jobs if you run: > > squeue -u csfoo -t pd,r,s,cg,cd,cf,ca,f,to,pr,bf,nf,rv,SPECIAL_EXIT > > I just want to make sure that the jobs aren't in a different state but still > somehow being counted against the users's limit. Sure: there doesn't seem to be any job in a different state when the problem occurs. For instance, here's the output right now: $ squeue -u csfoo -t pd,r,s,cg,cd,cf,ca,f,to,pr,bf,nf,rv,SPECIAL_EXIT JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 171780 akundaje sbatch csfoo R 0:29 1 sh-102-31 154557 akundaje submit_p csfoo R 1-21:33:39 1 sh-102-32 163042 akundaje submit_p csfoo R 22:08:35 1 sh-102-30 163043 akundaje submit_p csfoo R 22:08:35 1 sh-102-30 163044 akundaje submit_p csfoo R 22:08:35 1 sh-102-30 163040 akundaje submit_p csfoo R 22:08:38 1 sh-102-30 163041 akundaje submit_p csfoo R 22:08:38 1 sh-102-30 163038 akundaje submit_p csfoo R 22:08:41 1 sh-102-30 163039 akundaje submit_p csfoo R 22:08:41 1 sh-102-30 163035 akundaje submit_p csfoo R 22:08:43 1 sh-102-30 163036 akundaje submit_p csfoo R 22:08:43 1 sh-102-30 163037 akundaje submit_p csfoo R 22:08:43 1 sh-102-30 163034 akundaje submit_p csfoo R 22:08:46 1 sh-102-28 163032 akundaje submit_p csfoo R 22:08:49 1 sh-102-28 163033 akundaje submit_p csfoo R 22:08:49 1 sh-102-28 163029 akundaje submit_p csfoo R 22:08:53 1 sh-102-32 163030 akundaje submit_p csfoo R 22:08:53 1 sh-102-29 163031 akundaje submit_p csfoo R 22:08:53 1 sh-102-29 163028 akundaje submit_p csfoo R 22:08:56 1 sh-102-32 171259 akundaje sbatch csfoo R 2:26:15 1 sh-102-28 171262 akundaje sbatch csfoo R 2:25:25 1 sh-102-28 171702 akundaje sbatch csfoo R 1:12:56 1 sh-102-28 171699 akundaje sbatch csfoo R 1:17:38 1 sh-102-32 171690 akundaje sbatch csfoo R 1:29:10 1 sh-102-31 171654 akundaje sbatch csfoo R 1:55:25 1 sh-102-28 171695 akundaje sbatch csfoo R 1:22:37 1 sh-102-29 171754 akundaje sbatch csfoo R 39:59 1 sh-102-29 171749 akundaje sbatch csfoo R 44:48 1 sh-102-31 171718 akundaje sbatch csfoo R 56:46 1 sh-102-32 171716 akundaje sbatch csfoo R 57:14 1 sh-102-31 171771 akundaje sbatch csfoo R 17:14 1 sh-102-31 171760 akundaje sbatch csfoo R 33:09 1 sh-102-32 171759 akundaje sbatch csfoo R 33:30 1 sh-102-29 171774 akundaje sbatch csfoo R 13:05 1 sh-102-31 171777 akundaje sbatch csfoo R 3:23 1 sh-102-32 171755 akundaje sbatch csfoo R 39:50 1 sh-102-32 171050 normal sbatch csfoo R 5:28:20 1 sh-101-56 171693 normal sbatch csfoo R 1:24:33 1 sh-101-02 171691 normal sbatch csfoo R 1:28:14 1 sh-101-02 171720 normal sbatch csfoo R 54:07 1 sh-101-02 171776 normal sbatch csfoo R 8:40 1 sh-101-01 Plus, on the graph I attached earlier, we count the total number of jobs in queue, by parsing the output of "squeue -h -o '%g %u %P %b %T %C'". This should list all jobs, no matter what their state is, and as you can see on the graph, the absolute maximum total number of jobs barely reaches 1,600. So nowhere close to 3,000 for a single user. Thanks! -- Kilian Kilian,
I've been going over your slurm.conf file and I found one problem that may be causing the issue you're seeing. Looking at your partition definitions, I noticed that you're using both normal qos's and a partition qos. For example on the akundaje partition:
PartitionName=akundaje \
DefaultTime=2:0:0 \
DefMemPerCPU=12800 MaxMemPerCPU=12800 \ # 20c/256GB
PriorityTier=1000 \
AllowGroups=akundaje,sh_sysadm \
AllowQos=normal,owner,system \ <-------------------------
QOS=owner \ <-------------------------
nodes=sh-102-[28-32],sh-112-[06-07]
The problem here is that you're using "owner" as both a normal qos (AllowQos=) and a paritition qos (QOS=). Normally, "owner" would be listed on only one or the other. By listing it on both, you could be double-counting the usage in the owner partition. That would explain why your seeing a limit when you're approaching 1500 total jobs in the queue. I'm not 100% certain this is the cause of the problem but you'll want to to change it regardless. The easiest way would probably be to make another qos (e.g. "part_owner") that has the same limits as "owner" and use that as the partition qos. Like this:
PartitionName=akundaje \
DefaultTime=2:0:0 \
DefMemPerCPU=12800 MaxMemPerCPU=12800 \ # 20c/256GB
PriorityTier=1000 \
AllowGroups=akundaje,sh_sysadm \
AllowQos=normal,owner,system \
QOS=part_owner \ <-------------------------
nodes=sh-102-[28-32],sh-112-[06-07]
I recommend you make that change to all your partitions that list any qos more than once on both the "AllowQos=" and "QOS=" lines.
Let me know how that goes for you.
Regards.
Tim
Hi Tim,
Thanks for the suggestion. I somehow thought that the Partition QOS had to be listed in AllowQos for things to work, but obviously that was wrong.
I removed "owner" from the AllowQos list, so now, the partition definition looks like this
PartitionName=akundaje \
DefaultTime=2:0:0 \
DefMemPerCPU=12800 MaxMemPerCPU=12800 \ # 20c/256GB
PriorityTier=1000 \
AllowGroups=akundaje,sh_sysadm \
AllowQos=normal,system \
QOS=owner \
nodes=sh-102-[28-32],sh-112-[06-07]
I'll ask the user to resubmit jobs to see if the issue is still present.
But, to mitigate things a bit, I checked the job records and generated the same kind of graph for just that user, and he apparently never had more than ~400 jobs in queue. So even if the jobs were counted twice, that wouldn't account for the 3,00 job limit. (cf. attached graph).
Thanks a lot for looking into this!
Cheers,
--
Kilian
Created attachment 4695 [details]
job number history for user csfoo
Hi! Just another note to let you know that the user confirmed the behavior is still there after having removed QOS redundancy from the configuration. Cheers, -- Kilian Hi!
Another update: an other user, from a different group, reported the same issue. It happens on a different partition, although with a very similar configuration:
PartitionName=possu
AllowGroups=possu,sh_sysadm AllowAccounts=ALL AllowQos=normal,system
AllocNodes=ALL Default=NO QoS=owner
DefaultTime=02:00:00 DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO
MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=1 LLN=NO MaxCPUsPerNode=UNLIMITED
Nodes=sh-102-[17-21]
PriorityJobFactor=1 PriorityTier=1000 RootOnly=NO ReqResv=NO OverSubscribe=NO
OverTimeLimit=NONE PreemptMode=REQUEUE
State=UP TotalCPUs=100 TotalNodes=5 SelectTypeParameters=NONE
DefMemPerCPU=6400 MaxMemPerCPU=6400
The scheduler currently thinks that the user has 2,802 jobs in queue, when it really has none:
# squeue -u reguchi -t pd,r,s,cg,cd,cf,ca,f,to,pr,bf,nf,rv,SPECIAL_EXIT
JOBID PARTITION NAME USER ST TIME NODES
# su - reguchi
$ sbatch -p possu --array=1-200 --wrap="sleep 1000"
sbatch: error: Batch job submission failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)
The slurmctld logs:
slurmctld[85048]: job submit for user reguchi(294831): qos max submit job limit exceeded 3000, current jobs 2802, newly submitted job count 200
slurmctld[85048]: _job_create: exceeded association/qos's limit for user 294831
slurmctld[85048]: _slurm_rpc_submit_batch_job: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)
Any other information we could provide?
Thanks!
--
Kilian
And last thing FWIW, `scontrol reconfig` clears the "ghost" job list, and unblocks the situation. Cheers, -- Kilian Kilian - Tim's out on vacation this week, so I'm going to grab this from him. Can you get the output from 'scontrol show assoc' the next time you notice this? That should help isolate the cause. And did you restart slurmctld after ensuring that the QOS was not able to be used as both a PartitionQOS and normal QOS? - Tim Hi Tim, > Tim's out on vacation this week, so I'm going to grab this from him. Thanks! > Can you get the output from 'scontrol show assoc' the next time you notice > this? That should help isolate the cause. Sure, will do and report back. > And did you restart slurmctld after ensuring that the QOS was not able to be > used as both a PartitionQOS and normal QOS? Yes, slurmctld was restarted after the change. Cheers, -- Kilian Hi Tim, (In reply to Kilian Cavalotti from comment #22) > > Can you get the output from 'scontrol show assoc' the next time you notice > > this? That should help isolate the cause. > > Sure, will do and report back. Not exactly what was requested, but at least some clue that the accounting of jobs may be off. The user reported this discrepancy between the number of jobs he had in queue, and what "scontrol show assoc" reports: -- 8< -------------------------------------------------------------------- I was trying to reproduce the issue (was fine) and checked on the output of "scontrol show assoc", which suggests that I have 733 jobs submitted under the owner QOS, but I don't have nearly that many jobs in the queue (I had about 100 running jobs, which you can also see from the attached squeue output, and scontrol output at the beginning). User 36268 MaxJobsPU=N(44) MaxSubmitJobsPU=3000(733) MaxTRESPU=cpu=99999(91),mem=N(886603),energy=N(0),node=N(44),gres/gpu=N(0) Attached files were produced with: $ squeue -u csfoo -t all > squeue-t10.txt; $ scontrol show assoc > scontrol-assoc-t10.txt -- 8< -------------------------------------------------------------------- Outputs are attached. Thanks! -- Kilian Created attachment 4718 [details]
Output of "scontrol show assoc"
Created attachment 4719 [details]
Output of squeue
I should have asked for the slurm.conf earlier; I think I can readily explain this. Can you attach it for posterity? I'm guessing you're using that 'owner' QOS as a Partition QOS on multiple partitions? If so, then the usage from all of those is being summed together - think of the each QOS as a pool of resources / limits, and when used in multiple locations all that would sum together. I'm guessing you've applied the QOS to multiple partitions thinking it would track usage on each partition independently? If so, I think the fix here is to define separate QOS's, one per partition, and change each partition definition appropriately. (I usually recommend using part_ as a prefix to the QOS name to highlight the difference.) Created attachment 4720 [details]
slurm.conf
(In reply to Tim Wickberg from comment #26) > I should have asked for the slurm.conf earlier; I think I can readily > explain this. Can you attach it for posterity? Sure, done. > I'm guessing you're using that 'owner' QOS as a Partition QOS on multiple > partitions? Correct. > If so, then the usage from all of those is being summed together - think of > the each QOS as a pool of resources / limits, and when used in multiple > locations all that would sum together. Ah, I see. But there still something that confuses me. If you look back at the first comments on this bug, I posted a graph of the total number of jobs over a period of time where our user's jobs were rejected because of the 3,000 MaxSubmitPU limit. Except you can see that the total number of jobs across the whole cluster never exceeded ~1,600 jobs (the 2 graphs are stacked). So even if usage from all the partitions is summed together, it still doesn't reach the QOS's limit of 3,000 jobs. Isn't there still an accounting issue somewhere? > I'm guessing you've applied the QOS to multiple partitions thinking it would > track usage on each partition independently? If so, I think the fix here is > to define separate QOS's, one per partition, and change each partition > definition appropriately. (I usually recommend using part_ as a prefix to > the QOS name to highlight the difference.) I see how using separate QOSes for each partition would make sure the limits are applied to that partition only. But isn't that the whole point of a *partition* QOS already? The name kind of implies that the QOS limits would apply to the partition, independently of the other partitions. That's a bit confusing to me. Plus, we're regularly adding new partitions, as new groups join our condo cluster. So having multiple QOSes with identical limits seems a bit redundant. To take a step back, what we're trying to do is: * establish a set of default limits for our regular users, who can submit to a restricted set of partitions, * override those restrictions with higher default limits for our owners, when they submit jobs to their own partitions. And all of that without having the users to specify any QOS at submission time, nor using a job submission plugin. :) Anyway, of specific partition QOSes is the only way, we can live with it. But it still looks like there's something amiss with the counting of jobs, don't you think? Thanks! -- Kilian > So even if usage from all the partitions is summed together, it still > doesn't reach the QOS's limit of 3,000 jobs. > > Isn't there still an accounting issue somewhere? It's possible, but there may be a better explanation for that elsewhere I haven't found yet. If the "owner" QOS was being used as not just the partition QOS on all these condo partitions, but the QOS for the job itself, that may explain that odd result. > Plus, we're regularly adding new partitions, as new groups join our condo > cluster. So having multiple QOSes with identical limits seems a bit > redundant. Yeah - different way of thinking about it. From Slurm's perspective, "PartitionQOS" is all of the QOS capabilities applied to the partition rather than the individual jobs, and that includes the ability to be used across multiple partitions. Although I can see that's not obvious, and should be better documented. I'll think more on how to do that, and if there may be a way to add some warnings to the log files to hint at the potential complications. > To take a step back, what we're trying to do is: > * establish a set of default limits for our regular users, who can submit to > a restricted set of partitions, > * override those restrictions with higher default limits for our owners, > when they submit jobs to their own partitions. > And all of that without having the users to specify any QOS at submission > time, nor using a job submission plugin. :) > > Anyway, of specific partition QOSes is the only way, we can live with it. The recommendation I usually give as part of the training is, if you're going to use Partition QOSs, to set each of the QOS up as part_$partname through sacctmgr. As there's no internal differentiation between the QOS and partition QOSs that helps highlight the difference between them, and in your case I believe all of them would apply to a single partition. > But it still looks like there's something amiss with the counting of jobs, > don't you think? Let's get the QOS cleared up, and see if you notice that again. I haven't seen an easy path into reproducing that behavior, or anything particularly unusual in the log files thus far. (In reply to Tim Wickberg from comment #29) > > So even if usage from all the partitions is summed together, it still > > doesn't reach the QOS's limit of 3,000 jobs. > > > > Isn't there still an accounting issue somewhere? > > It's possible, but there may be a better explanation for that elsewhere I > haven't found yet. > > If the "owner" QOS was being used as not just the partition QOS on all these > condo partitions, but the QOS for the job itself, that may explain that odd > result. Jobs were submitted with the following command: $ srun -J sdev -t 01:00:00 -p akundaje --mem 4GB --pty bash ie. without explicitly requesting a QOS, so I'm assuming the default QOS (normal) was used: # sacctmgr show assoc where user=csfoo format=user,qos,defaultqos User QOS Def QOS ---------- -------------------- --------- csfoo long,normal normal > The recommendation I usually give as part of the training is, if you're > going to use Partition QOSs, to set each of the QOS up as part_$partname > through sacctmgr. As there's no internal differentiation between the QOS and > partition QOSs that helps highlight the difference between them, and in your > case I believe all of them would apply to a single partition. Gotcha, I'll do that. I was so hoping of getting rid of the many similar QOSes we've been using so far. :) > Let's get the QOS cleared up, and see if you notice that again. I haven't > seen an easy path into reproducing that behavior, or anything particularly > unusual in the log files thus far. Sounds good, thanks! Cheers, -- Kilian (In reply to Kilian Cavalotti from comment #30) > Gotcha, I'll do that. I was so hoping of getting rid of the many similar > QOSes we've been using so far. :) > > > Let's get the QOS cleared up, and see if you notice that again. I haven't > > seen an easy path into reproducing that behavior, or anything particularly > > unusual in the log files thus far. > > Sounds good, thanks! All right, I think I'm convinced now. I was able to reproduce the problem by submitting jobs to 3 different partitions that use the same Partition QOS (with a MaxJobsPU of 3,000): after submitting 1,000 jobs to each of those 3 partitions, I couldn't submit another job to any of them, and was receiving the "Job violates accounting/QOS policy" message. So I've implemented the solution you suggested: specific partition QOSes for each partition. I still feel this could be made more explicit in the documentation, because I don't think I'll the only one to believe that partition QOS limits only apply to a single partition. Thanks for the clarification and suggestions, as always! Cheers, -- Kilian Hi Tim, So, although I still understand that the previous situation of sharing the same partition QOS for multiple partitions was a recipe for limits snafu, there still seems to be something amiss. We now have partition-specific partition QOSes (quite the mouthful), and still, "scontrol show assoc" reports more the jobs than users are actually submitting. I'm attaching new "squeue" and "scontrol show assoc" outputs, that shows that user "csfoo" has 74 jobs in queue, while the scheduler thinks he has 98. Files generated with: $ squeue -u csfoo -t all > squeue-t11.txt; scontrol show assoc > scontrol-assoc-t11.txt Any insight? Cheers, -- Kilian Created attachment 4728 [details]
squeue-t11.txt
Created attachment 4729 [details]
scontrol-assoc-t11.txt
Hi SchedMD! New update about the job count discrepancies between "squeue" and "scontrol show assoc". Here's what our user reported: -- 8< -------------------------------------------------------------------- Haven't been able to reproduce the exact issue again, but here's a more dramatic example of what seems to be an accounting curiosity - I have 9 jobs in part_kundaje QOS but it thinks I have 353 submitted, and the numbers for the GrpSubmitJobs also don't seem to add up. As usual, outputs of squeue and scontrol are attached, generated with squeue -u csfoo -t all > squeue-t13.txt; scontrol show assoc > scontrol-assoc-t13.txt QOS=part_akundaje(132) UsageRaw=12485377.071585 GrpJobs=N(33) GrpSubmitJobs=N(205) GrpWall=N(86483.82) GrpTRES=cpu=N(113),mem=N(692317),energy=N(0),node=N(33),gres/gpu=N(0) GrpTRESMins=cpu=N(208089),mem=N(2038367787),energy=N(0),node=N(86483),gres/gpu=N(0) GrpTRESRunMins=cpu=N(214331),mem=N(2243087158),energy=N(0),node=N(90514),gres/gpu=N(0) MaxWallPJ=10080 MaxTRESPJ= MaxTRESPN= MaxTRESMinsPJ= MinTRESPJ= PreemptMode=OFF Priority=0 Account Limits Account akundaje MaxJobsPA=N(33) MaxSubmitJobsPA=5000(205) MaxTRESPA=cpu=99999(113),mem=N(692317),energy=N(0),node=N(33),gres/gpu=N(0) User Limits User 36268 MaxJobsPU=N(9) MaxSubmitJobsPU=3000(353) MaxTRESPU=cpu=99999(17),mem=N(212317),energy=N(0),node=N(9),gres/gpu=N(0) User 312359 MaxJobsPU=N(24) MaxSubmitJobsPU=3000(0) MaxTRESPU=cpu=99999(96),mem=N(480000),energy=N(0),node=N(24),gres/gpu=N(0) -- 8< -------------------------------------------------------------------- Created attachment 4747 [details]
squeue-t13.txt
Created attachment 4748 [details]
scontrol-assoc-t13.txt
Hi SchedMD! Monday update: the issue is still here. Despite now having distinct QOSes for each partition, the user still managed to reproduce the issue where, after some time, his job submissions are rejected with "Job violates accounting/QOS policy" Here are the commands used to generate the "squeue" and "scontrol show assoc" outputs that are gonna be attached: -- 8< --------------------------------------------------------------------- $ squeue -u csfoo -t all > squeue-t16.txt; scontrol show assoc > scontrol-assoc-t16.txt $ /usr/bin/srun -J sdev -t 24:00:00 -p akundaje --mem 4GB --pty /bin/bash -i -l srun: error: Unable to allocate resources: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits) -- 8< --------------------------------------------------------------------- User Limits User 36268 MaxJobsPU=N(4) MaxSubmitJobsPU=3000(3000) MaxTRESPU=cpu=99999(10),mem=N(89688),energy=N(0),node=N(4),gres/gpu=N(0) We would appreciate any update if you got a chance. Thanks! -- Kilian Created attachment 4756 [details]
squeue-t16.txt
Created attachment 4757 [details]
scontrol-assoc-t16.txt
Hi SchedMD! Just a friendly ping to see if you had any chance to look into this. We're still experiencing this issue, and it's starting to be blocking for our users. Thanks! -- Kilian Hi!
I'm taking the liberty to bump the Importance of that bug, since over the week-end, several other users reported the issue. It seems to happen on different partitions, using different QOSes, and for different users.
For instance, user "possu(306840)" has 0 (zero) job in queue, and yet, Slurm considers it submitted 2638:
# squeue -u possu
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
# scontrol show assoc
QOS=owners(58)
UsageRaw=935234674.078245
GrpJobs=N(9) GrpSubmitJobs=N(2613) GrpWall=N(1656485.69)
GrpTRES=cpu=N(540),mem=N(3936000),energy=N(0),node=N(29),gres/gpu=N(0)
GrpTRESMins=cpu=N(15587056),mem=N(92845143570),energy=N(0),node=N(2215394),gres/gpu=N(0)
GrpTRESRunMins=cpu=N(1168127),mem=N(8639433733),energy=N(0),node=N(64115),gres/gpu=N(0)
MaxWallPJ=2880
MaxTRESPJ=
MaxTRESPN=
MaxTRESMinsPJ=
MinTRESPJ=
PreemptMode=OFF
Priority=0
[...]
Account Limits
[...]
Account possu
MaxJobsPA=N(0) MaxSubmitJobsPA=5000(2638)
MaxTRESPA=cpu=4096(0),mem=N(0),energy=N(0),node=N(0),gres/gpu=N(0)
[...]
User Limits
[...]
User 306840
MaxJobsPU=N(0)
MaxSubmitJobsPU=3000(2638)
MaxTRESPU=cpu=2048(0),mem=N(0),energy=N(0),node=N(0),gres/gpu=N(0)
I'll attach the full output of "squeue" and "scontrol show assoc"
Please let us know if we can provide any more information to resolve this issue. As I mentioned, this is getting more and more pressing, as it potentially affects every user and prevent them to submit jobs after some time.
Thanks!
--
Kilian
Created attachment 4789 [details]
scontrol-assoc.txt
Created attachment 4790 [details]
squeue.txt
Sorry, Killian, I'll see what I can find on this. Killian, in your last report from comments 43-45 you will find
ClusterName=sherlock Account=possu UserName=parberg(294523) Partition= ID=6940
SharesRaw/Norm/Level/Factor=100/0.12/800/0.05
UsageRaw/Norm/Efctv=6866827.41/0.00/0.07
ParentAccount= Lft=5406 DefAssoc=Yes
GrpJobs=N(128)
GrpSubmitJobs=N(600) GrpWall=N(114447.51)
GrpTRES=cpu=N(128),mem=N(512000),energy=N(0),node=N(128),gres/gpu=N(0)
GrpTRESMins=cpu=N(114447),mem=N(457788493),energy=N(0),node=N(114447),gres/gpu=N(0)
GrpTRESRunMins=cpu=N(129246),mem=N(516987600),energy=N(0),node=N(129246),gres/gpu=N(0)
MaxJobs= MaxSubmitJobs= MaxWallPJ=
MaxTRESPJ=
MaxTRESPN=
MaxTRESMinsPJ=
Is the one using up resources for account possu (not the user possu).
Looking at (what I am guessing is the job's QOS 'normal')
QOS=normal(3)
...
Account possu
MaxJobsPA=N(128) MaxSubmitJobsPA=3000(600)
MaxTRESPA=cpu=256(128),mem=N(512000),energy=N(0),node=N(128),gres/gpu=N(0)
Could you send me the output of
scontrol show job 275581 (or 275582, 275583, 275584)
I am guessing they are asking for more than 128 cpus on those jobs?
Hi Dany, Thanks for the feedback, much appreciated. (In reply to Danny Auble from comment #46) > Killian, in your last report from comments 43-45 you will find > > ClusterName=sherlock Account=possu UserName=parberg(294523) Partition= > ID=6940 > GrpSubmitJobs=N(600) GrpWall=N(114447.51) > GrpTRES=cpu=N(128),mem=N(512000),energy=N(0),node=N(128),gres/gpu=N(0) > Is the one using up resources for account possu (not the user possu). "parberg" has jobs running indeed, but only 129, not the 2,600+ that are reported by "scontrol show assoc" for user "possu": # grep -c parberg squeue.txt 129 > Could you send me the output of > scontrol show job 275581 (or 275582, 275583, 275584) Sure (attached in jobs.txt). > I am guessing they are asking for more than 128 cpus on those jobs? No, they apparently each request 1 node, 16 CPUs. And I'm confused: the limit being hit is MaxSubmitJobsPerUser (cf. instrumented logs in https://bugs.schedmd.com/attachment.cgi?id=4690), so the number of CPUs shouldn't be involved, right? Thanks for looking into it! Cheers, -- Kilian Created attachment 4791 [details]
scontrol show job for 275581 275582 275583 275584
(In reply to Kilian Cavalotti from comment #47 > No, they apparently each request 1 node, 16 CPUs. > > And I'm confused: the limit being hit is MaxSubmitJobsPerUser (cf. > instrumented logs in https://bugs.schedmd.com/attachment.cgi?id=4690), so > the number of CPUs shouldn't be involved, right? I was only going on squeue, those jobs list QOSMaxCpuPerUserLimit. I notice 2 of those jobs I requested information or did start running. I am guessing these are not related. I did notice a potential issue though related to what you are seeing. I will attach a patch in a moment. You wouldn't happen to know if csfoo is submitting array's that are larger than 3000? Created attachment 4792 [details]
Patch to correctly remove jobs from qos submitted jobs
Killian, please try this patch and report back if things correct themselves.
Before trying this patch (and assuming the counts are still in question) please restart the slurmctld and see if it fixes things by just restarting.
(In reply to Danny Auble from comment #49) > I was only going on squeue, those jobs list QOSMaxCpuPerUserLimit. I notice > 2 of those jobs I requested information or did start running. I am guessing > these are not related. Oh yeah, ok. We indeed have cpu=128 MaxTRESPU in the "normal" QOS. The blocking we're seeing is about MaxSubmitPU in the "owners" and "part_<owner>" partition QOSes. (I know, the ticket history is a bit long, sorry...) > I did notice a potential issue though related to what you are seeing. I > will attach a patch in a moment. Great! > You wouldn't happen to know if csfoo is submitting array's that are larger > than 3000? I know he's not, because: # scontrol show config | grep MaxArray MaxArraySize = 1001 Thanks! -- Kilian (In reply to Danny Auble from comment #50) > Created attachment 4792 [details] > Patch to correctly remove jobs from qos submitted jobs > > Killian, please try this patch and report back if things correct themselves. Will do, thanks. > Before trying this patch (and assuming the counts are still in question) > please restart the slurmctld and see if it fixes things by just restarting. Yes, restarting slurmctld has been confirmed to clear up the "ghost jobs" counts and unblock the situation for a time, until it happens again. Will try the patch right now. Cheers, -- Kilian Awesome. This code was definitely wrong before, as it is directly related to the problem you are seeing I am hoping this will fix the situation :). (In reply to Danny Auble from comment #53) > Awesome. This code was definitely wrong before, as it is directly related > to the problem you are seeing I am hoping this will fix the situation :). Great, me too! Patch is deployed, I will get back in touch with the users to ask them to submit jobs and will report back. Thanks! -- Kilian Killian, I just reproduced the issue. It has to deal with job's moving partitions while pending. It doesn't look like the accounting was thought out very well for this situation. I'll see if I can get a patch that will handle at least the simple scenarios. I'll let you know what I come up with. I will make note the last patch (now commit a034e4ab8080) is an improvement to the code, but will not solve the scenario. (In reply to Danny Auble from comment #55) > Killian, I just reproduced the issue. It has to deal with job's moving > partitions while pending. It doesn't look like the accounting was thought > out very well for this situation. > > I'll see if I can get a patch that will handle at least the simple > scenarios. I'll let you know what I come up with. > > I will make note the last patch (now commit a034e4ab8080) is an improvement > to the code, but will not solve the scenario. Aaah, so that would be the case for jobs submitted to multiple partitions, right? Our users are especially fond of those... Cheers, -- Kilian (In reply to Kilian Cavalotti from comment #56) > Aaah, so that would be the case for jobs submitted to multiple partitions, > right? Our users are especially fond of those... And I just confirmed that all the users affected by this were indeed submitting jobs to multiple partitions. Good catch! Cheers, -- Kilian You got it. Or if someone changed the partition with scontrol. Glad you were able to confirm :). One note, behavior will change here on job submit. If the user breaks any partition's QOS MaxSubmit* limit they will be deigned from running. Also note if you have the same QOS as the partition QOS the submit count will go up multiple times. For instance PartitionName=part1 qos=part_qos PartitionName=part2 qos=part_qos sbatch -ppart1,part2 Then the count for part_qos would be MaxSubmitJobsPA=N(2), making it possible to break a limit of 10 with 1 job if you had the same qos in 11 different partitions. Does that make sense? I don't know if you do it this way (not advised) but I have seen some people do do it this way ;). Created attachment 4793 [details]
Patch to handle multiple partitions that have a partition QOS for submit limits
Killian, I believe this patch will fix the scenario. I have tested if fairly well, but when you use it I would monitor the output of scontrol show assoc every once in a while to make sure nothing looks totally out of place.
What this patch does is different than I explained before in respects to a job counting multiple times if the partition qos was the same for multiple partitions. It does not do that, it only counts it once no matter how many partitions the job is submitted to.
In any case. Please let me know how this looks on your system, if you see anything out of the ordinary or not. We will try to get this into 17.02.5 (due this week).
Thanks for your patience on the matter.
Created attachment 4794 [details]
Patch to handle multiple partitions that have a partition QOS for submit limits
We just found a bad line (shouldn't cause any real issue). I just fixed it in this patch.
List part_qos_list = part_qos_list = list_create(NULL);
No reason to do that double set huh ;).
(In reply to Danny Auble from comment #61) > Created attachment 4794 [details] > Patch to handle multiple partitions that have a partition QOS for submit > limits > > We just found a bad line (shouldn't cause any real issue). I just fixed it > in this patch. > > List part_qos_list = part_qos_list = list_create(NULL); > > No reason to do that double set huh ;). Awesome, thanks Danny! Patch #2 applied and deployed, will monitor things and report back. Cheers, -- Kilian Just a quick update to say that things look good so far: "scontrol show assoc" has been consistently accurate in reporting the number of jobs in queue for each user. So, I'll wait a few more days, but that looks very promising, thanks! Cheers, -- Kilian Fantasic, we did find one minor issue that it doesn't appear your environment has. It has to deal with no partitions the job was submitted to not having any qos. If all your partitions have a QOS then the difference wouldn't matter. The final patch is in commit 4b003d37ab5 and will be in 17.02.5 most likely tagged later this week. Kilian, please reopen this if you still experience issues. I think this one is solved though :). (In reply to Danny Auble from comment #70) > Kilian, please reopen this if you still experience issues. I think this one > is solved though :). Great job in fixing this, thank you! Cheers, -- Kilian Hello. We would like this fix in the next version of 16.05 (https://github.com/SchedMD/slurm/commit/4b003d37ab5) Thanks. I am sorry, there is no current plan to release another 16.05. I would suggest locally backporting the patch until you are able to upgrade to 17.02+. Thanks! No bother :). Unfortunately this will not be backported. You will still have to locally backporting the patch until you are able to upgrade to 17.02+. Note, support for 16.05 will end in ~2 months. Thanks! |