Hello Killian,

I tried to go to https://stanford.box.com/v/slurmconf to see the whole slurm.conf but it requires a login and password.

Can you just attach it to the bug?

Thanks

Tim

Hi Tim,  

My bad. It should be better now, if you want to take another shot at it.

Thanks,
-- 
Kilian

Created attachment 4663 [details]
Logging patch

I put together a logging patch that logs a few more things at the info level to help make it apparent why the controller thinks your user is over their limit.  Please apply the logging patch and have the user reproduce the error.  Then paste in the new log message.  This is the log line that I added a few new fields to:

job submit for user csfoo(36268): qos max submit job limit exceeded 3000, current jobs <>, newly submitted job count <>

The first number is the limit (3000).  The second number is the current number of jobs (pending or running) and the third is the number of jobs in the job submission (usually 1 unless it's an array job).

Also if you could paste in the output of:

squeue -u csfoo

That would be helpful.

Thanks

Tim

Hi Tim, 

(In reply to Tim Shaw from comment #4)
> I put together a logging patch that logs a few more things at the info level
> to help make it apparent why the controller thinks your user is over their
> limit.  Please apply the logging patch and have the user reproduce the
> error.  

Thanks, I'll work on that. And provide the squeue output too.

Cheers,
-- 
Kilian

Hi Tim, 

Sorry for the delay, the issue only manifests itself after a bunch of jobs have been submitted, typically 12-24 hours.

Here is what the patched slurmctld logged when this happened again:

-- 8< ---------------------------------
Jun  2 04:19:43 sh-sl01 slurmctld[85048]: job submit for user csfoo(36268): qos max submit job limit exceeded 3000, current jobs 3000, newly submitted job count 1
Jun  2 04:19:43 sh-sl01 slurmctld[85048]: _job_create: exceeded association/qos's limit for user 36268
Jun  2 04:19:43 sh-sl01 slurmctld[85048]: _slurm_rpc_submit_batch_job: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)
-- 8< ---------------------------------

So it looks like the scheduler thinks there are 3000 jobs from that user in queue at that time, but there hasn't been. I'm attaching both the squeue output for that user (squeue -u csfoo) at the time this happened, and a graph of the total number of pending and running jobs for the whole cluster.


Here's an excerpt of the job submission tries from that user (t8.txt is the attached squeue output):

-- 8< ---------------------------------
2017-06-02 04:19:55 [csfoo@sh-ln02]$ /usr/bin/srun -J sdev -t 01:00:00 -p akundaje --mem 4GB --pty /bin/bash -i -l 
srun: error: Unable to allocate resources: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits) 
2017-06-02 04:19:58 [csfoo@sh-ln02]$ /usr/bin/srun -J sdev -t 01:00:00 -p akundaje --mem 4GB --pty /bin/bash -i -l 
srun: error: Unable to allocate resources: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits) 
2017-06-02 04:19:59 [csfoo@sh-ln02]$ /usr/bin/srun -J sdev -t 01:00:00 -p akundaje --mem 4GB --pty /bin/bash -i -l 
srun: error: Unable to allocate resources: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits) 
2017-06-02 04:20:00 [csfoo@sh-ln02]$ squeue -u csfoo --Format=jobid,partition,name,state,timeused,timelimit,numnodes,numcpus,nodelist > t8.txt 
-- 8< ---------------------------------

Then apparently, when he tried to submit a job to multiple partitions, it "unblocked" the situation:

-- 8< ---------------------------------
2017-06-02 04:25:55 [csfoo@sh-ln02]$ /usr/bin/srun -J sdev -t 01:00:00 -p akundaje --mem 4GB --pty /bin/bash -i -l
srun: error: Unable to allocate resources: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits) 
2017-06-02 04:25:57 [csfoo@sh-ln02]$ /usr/bin/srun -J sdev -t 01:00:00 -p owners,akundaje --mem 4GB --pty /bin/bash -i -l 
srun: job 170981 queued and waiting for resources 
srun: job 170981 has been allocated resources 
[csfoo@sh-102-31]$ exit 
logout 
2017-06-02 04:26:13 [csfoo@sh-ln02]$ /usr/bin/srun -J sdev -t 01:00:00 -p akundaje --mem 4GB --pty /bin/bash -i -l 
[csfoo@sh-102-31]$ exit 
logout 
2017-06-02 04:26:19 [csfoo@sh-ln02]$
-- 8< ---------------------------------

I'm also attaching the full slurmctld.log for that time period.

Thanks!
-- 
Kilian

Created attachment 4690 [details]
slurmctld.log

Created attachment 4691 [details]
Number of jobs

That graph shows the total number of pending and running jobs on the whole cluster over the last 24 hours. It never reached 3,000 globally, let alone for a single user.

Created attachment 4692 [details]
squeue output

Killian,

I'm still investigating this problem but I wanted to know if you see any different squeue results with the users's jobs if you run:

squeue -u csfoo -t pd,r,s,cg,cd,cf,ca,f,to,pr,bf,nf,rv,SPECIAL_EXIT

I just want to make sure that the jobs aren't in a different state but still somehow being counted against the users's limit.

Thanks

Tim

(In reply to Tim Shaw from comment #12)
> I'm still investigating this problem but I wanted to know if you see any
> different squeue results with the users's jobs if you run:
> 
> squeue -u csfoo -t pd,r,s,cg,cd,cf,ca,f,to,pr,bf,nf,rv,SPECIAL_EXIT
> 
> I just want to make sure that the jobs aren't in a different state but still
> somehow being counted against the users's limit.

Sure: there doesn't seem to be any job in a different state when the problem occurs. For instance, here's the output right now:

$ squeue -u csfoo -t pd,r,s,cg,cd,cf,ca,f,to,pr,bf,nf,rv,SPECIAL_EXIT
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
            171780  akundaje   sbatch    csfoo  R       0:29      1 sh-102-31
            154557  akundaje submit_p    csfoo  R 1-21:33:39      1 sh-102-32
            163042  akundaje submit_p    csfoo  R   22:08:35      1 sh-102-30
            163043  akundaje submit_p    csfoo  R   22:08:35      1 sh-102-30
            163044  akundaje submit_p    csfoo  R   22:08:35      1 sh-102-30
            163040  akundaje submit_p    csfoo  R   22:08:38      1 sh-102-30
            163041  akundaje submit_p    csfoo  R   22:08:38      1 sh-102-30
            163038  akundaje submit_p    csfoo  R   22:08:41      1 sh-102-30
            163039  akundaje submit_p    csfoo  R   22:08:41      1 sh-102-30
            163035  akundaje submit_p    csfoo  R   22:08:43      1 sh-102-30
            163036  akundaje submit_p    csfoo  R   22:08:43      1 sh-102-30
            163037  akundaje submit_p    csfoo  R   22:08:43      1 sh-102-30
            163034  akundaje submit_p    csfoo  R   22:08:46      1 sh-102-28
            163032  akundaje submit_p    csfoo  R   22:08:49      1 sh-102-28
            163033  akundaje submit_p    csfoo  R   22:08:49      1 sh-102-28
            163029  akundaje submit_p    csfoo  R   22:08:53      1 sh-102-32
            163030  akundaje submit_p    csfoo  R   22:08:53      1 sh-102-29
            163031  akundaje submit_p    csfoo  R   22:08:53      1 sh-102-29
            163028  akundaje submit_p    csfoo  R   22:08:56      1 sh-102-32
            171259  akundaje   sbatch    csfoo  R    2:26:15      1 sh-102-28
            171262  akundaje   sbatch    csfoo  R    2:25:25      1 sh-102-28
            171702  akundaje   sbatch    csfoo  R    1:12:56      1 sh-102-28
            171699  akundaje   sbatch    csfoo  R    1:17:38      1 sh-102-32
            171690  akundaje   sbatch    csfoo  R    1:29:10      1 sh-102-31
            171654  akundaje   sbatch    csfoo  R    1:55:25      1 sh-102-28
            171695  akundaje   sbatch    csfoo  R    1:22:37      1 sh-102-29
            171754  akundaje   sbatch    csfoo  R      39:59      1 sh-102-29
            171749  akundaje   sbatch    csfoo  R      44:48      1 sh-102-31
            171718  akundaje   sbatch    csfoo  R      56:46      1 sh-102-32
            171716  akundaje   sbatch    csfoo  R      57:14      1 sh-102-31
            171771  akundaje   sbatch    csfoo  R      17:14      1 sh-102-31
            171760  akundaje   sbatch    csfoo  R      33:09      1 sh-102-32
            171759  akundaje   sbatch    csfoo  R      33:30      1 sh-102-29
            171774  akundaje   sbatch    csfoo  R      13:05      1 sh-102-31
            171777  akundaje   sbatch    csfoo  R       3:23      1 sh-102-32
            171755  akundaje   sbatch    csfoo  R      39:50      1 sh-102-32
            171050    normal   sbatch    csfoo  R    5:28:20      1 sh-101-56
            171693    normal   sbatch    csfoo  R    1:24:33      1 sh-101-02
            171691    normal   sbatch    csfoo  R    1:28:14      1 sh-101-02
            171720    normal   sbatch    csfoo  R      54:07      1 sh-101-02
            171776    normal   sbatch    csfoo  R       8:40      1 sh-101-01


Plus, on the graph I attached earlier, we count the total number of jobs in queue, by parsing the output of "squeue -h -o '%g %u %P %b %T %C'". This should list all jobs, no matter what their state is, and as you can see on the graph, the absolute maximum total number of jobs barely reaches 1,600. So nowhere close to 3,000 for a single user.

Thanks!
-- 
Kilian

Kilian,

I've been going over your slurm.conf file and I found one problem that may be causing the issue you're seeing.  Looking at your partition definitions, I noticed that you're using both normal qos's and a partition qos.  For example on the akundaje partition:

PartitionName=akundaje \
    DefaultTime=2:0:0 \
    DefMemPerCPU=12800  MaxMemPerCPU=12800 \ # 20c/256GB
    PriorityTier=1000 \
    AllowGroups=akundaje,sh_sysadm \
    AllowQos=normal,owner,system \          <-------------------------
    QOS=owner \                             <-------------------------
        nodes=sh-102-[28-32],sh-112-[06-07]

The problem here is that you're using "owner" as both a normal qos (AllowQos=) and a paritition qos (QOS=).  Normally, "owner" would be listed on only one or the other.  By listing it on both, you could be double-counting the usage in the owner partition.  That would explain why your seeing a limit when you're approaching 1500 total jobs in the queue.  I'm not 100% certain this is the cause of the problem but you'll want to to change it regardless.  The easiest way would probably be to make another qos (e.g. "part_owner") that has the same limits as "owner" and use that as the partition qos.  Like this:

PartitionName=akundaje \
    DefaultTime=2:0:0 \
    DefMemPerCPU=12800  MaxMemPerCPU=12800 \ # 20c/256GB
    PriorityTier=1000 \
    AllowGroups=akundaje,sh_sysadm \
    AllowQos=normal,owner,system \
    QOS=part_owner \                        <-------------------------
        nodes=sh-102-[28-32],sh-112-[06-07]

I recommend you make that change to all your partitions that list any qos more than once on both the "AllowQos=" and "QOS=" lines.

Let me know how that goes for you.

Regards.

Tim

Hi Tim, 

Thanks for the suggestion. I somehow thought that the Partition QOS had to be listed in AllowQos for things to work, but obviously that was wrong.

I removed "owner" from the AllowQos list, so now, the partition definition looks like this 

PartitionName=akundaje \                                                        
    DefaultTime=2:0:0 \                                                         
    DefMemPerCPU=12800  MaxMemPerCPU=12800 \ # 20c/256GB                        
    PriorityTier=1000 \                                                         
    AllowGroups=akundaje,sh_sysadm \                                            
    AllowQos=normal,system \                                                    
    QOS=owner \                                                                 
        nodes=sh-102-[28-32],sh-112-[06-07]


I'll ask the user to resubmit jobs to see if the issue is still present.

But, to mitigate things a bit, I checked the job records and generated the same kind of graph for just that user, and he apparently never had more than ~400 jobs in queue. So even if the jobs were counted twice, that wouldn't account for the 3,00 job limit. (cf. attached graph).

Thanks a lot for looking into this!

Cheers,
-- 
Kilian

Created attachment 4695 [details]
job number history for user csfoo

Hi!

Just another note to let you  know that the user confirmed the behavior is still there after having removed QOS redundancy from the configuration.

Cheers,
-- 
Kilian

Hi!

Another update: an other user, from a different group, reported the same issue. It happens on a different partition, although with a very similar configuration:

PartitionName=possu
   AllowGroups=possu,sh_sysadm AllowAccounts=ALL AllowQos=normal,system
   AllocNodes=ALL Default=NO QoS=owner
   DefaultTime=02:00:00 DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO
   MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=1 LLN=NO MaxCPUsPerNode=UNLIMITED
   Nodes=sh-102-[17-21]
   PriorityJobFactor=1 PriorityTier=1000 RootOnly=NO ReqResv=NO OverSubscribe=NO
   OverTimeLimit=NONE PreemptMode=REQUEUE
   State=UP TotalCPUs=100 TotalNodes=5 SelectTypeParameters=NONE
   DefMemPerCPU=6400 MaxMemPerCPU=6400


The scheduler currently thinks that the user has 2,802 jobs in queue, when it really has none:

# squeue -u reguchi -t pd,r,s,cg,cd,cf,ca,f,to,pr,bf,nf,rv,SPECIAL_EXIT
             JOBID PARTITION     NAME     USER ST       TIME  NODES 

# su - reguchi
$ sbatch -p possu --array=1-200 --wrap="sleep 1000"
sbatch: error: Batch job submission failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)

The slurmctld logs:

slurmctld[85048]: job submit for user reguchi(294831): qos max submit job limit exceeded 3000, current jobs 2802, newly submitted job count 200
slurmctld[85048]: _job_create: exceeded association/qos's limit for user 294831
slurmctld[85048]: _slurm_rpc_submit_batch_job: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)
 

Any other information we could provide?

Thanks!
-- 
Kilian

And last thing FWIW, `scontrol reconfig` clears the "ghost" job list, and unblocks the situation.

Cheers,
-- 
Kilian

Kilian -

Tim's out on vacation this week, so I'm going to grab this from him.

Can you get the output from 'scontrol show assoc' the next time you notice this? That should help isolate the cause.

And did you restart slurmctld after ensuring that the QOS was not able to be used as both a PartitionQOS and normal QOS?

- Tim

Hi Tim, 

> Tim's out on vacation this week, so I'm going to grab this from him.

Thanks!

> Can you get the output from 'scontrol show assoc' the next time you notice
> this? That should help isolate the cause.

Sure, will do and report back.

> And did you restart slurmctld after ensuring that the QOS was not able to be
> used as both a PartitionQOS and normal QOS?

Yes, slurmctld was restarted after the change.

Cheers,
-- 
Kilian

Hi Tim, 

(In reply to Kilian Cavalotti from comment #22)
> > Can you get the output from 'scontrol show assoc' the next time you notice
> > this? That should help isolate the cause.
> 
> Sure, will do and report back.

Not exactly what was requested, but at least some clue that the accounting of jobs may be off. The user reported this discrepancy between the number of jobs he had in queue, and what "scontrol show assoc" reports:

-- 8< --------------------------------------------------------------------
I was trying to reproduce the issue (was fine) and checked on the output of "scontrol show assoc", which suggests that I have 733 jobs submitted under the owner QOS, but I don't have nearly that many jobs in the queue (I had about 100 running jobs, which you can also see from the attached squeue output, and scontrol output at the beginning).

User 36268 
MaxJobsPU=N(44) 
MaxSubmitJobsPU=3000(733) 
MaxTRESPU=cpu=99999(91),mem=N(886603),energy=N(0),node=N(44),gres/gpu=N(0)

Attached files were produced with: 
$ squeue -u csfoo -t all > squeue-t10.txt; 
$ scontrol show assoc > scontrol-assoc-t10.txt
-- 8< --------------------------------------------------------------------

Outputs are attached.

Thanks!
-- 
Kilian

Created attachment 4718 [details]
Output of "scontrol show assoc"

Created attachment 4719 [details]
Output of squeue

I should have asked for the slurm.conf earlier; I think I can readily explain this. Can you attach it for posterity?

I'm guessing you're using that 'owner' QOS as a Partition QOS on multiple partitions?

If so, then the usage from all of those is being summed together - think of the each QOS as a pool of resources / limits, and when used in multiple locations all that would sum together.

I'm guessing you've applied the QOS to multiple partitions thinking it would track usage on each partition independently? If so, I think the fix here is to define separate QOS's, one per partition, and change each partition definition appropriately. (I usually recommend using part_ as a prefix to the QOS name to highlight the difference.)

Created attachment 4720 [details]
slurm.conf

(In reply to Tim Wickberg from comment #26)
> I should have asked for the slurm.conf earlier; I think I can readily
> explain this. Can you attach it for posterity?

Sure, done.

> I'm guessing you're using that 'owner' QOS as a Partition QOS on multiple
> partitions?

Correct.

> If so, then the usage from all of those is being summed together - think of
> the each QOS as a pool of resources / limits, and when used in multiple
> locations all that would sum together.

Ah, I see. 

But there still something that confuses me. If you look back at the first comments on this bug, I posted a graph of the total number of jobs over a period of time where our user's jobs were rejected because of the 3,000 MaxSubmitPU limit. Except you can see that the total number of jobs across the whole cluster never exceeded ~1,600 jobs (the 2 graphs are stacked).

So even if usage from all the partitions is summed together, it still doesn't reach the QOS's limit of 3,000 jobs. 

Isn't there still an accounting issue somewhere?

> I'm guessing you've applied the QOS to multiple partitions thinking it would
> track usage on each partition independently? If so, I think the fix here is
> to define separate QOS's, one per partition, and change each partition
> definition appropriately. (I usually recommend using part_ as a prefix to
> the QOS name to highlight the difference.)

I see how using separate QOSes for each partition would make sure the limits are applied to that partition only. But isn't that the whole point of a *partition* QOS already? The name kind of implies that the QOS limits would apply to the partition, independently of the other partitions. That's a bit confusing to me.

Plus, we're regularly adding new partitions, as new groups join our condo cluster. So having multiple QOSes with identical limits seems a bit redundant.


To take a step back, what we're trying to do is:
* establish a set of default limits for our regular users, who can submit to a restricted set of partitions,
* override those restrictions with higher default limits for our owners, when they submit jobs to their own partitions.
And all of that without having the users to specify any QOS at submission time, nor using a job submission plugin. :)

Anyway, of specific partition QOSes is the only way, we can live with it. But it still looks like there's something amiss with the counting of jobs, don't you think?

Thanks!
-- 
Kilian

> So even if usage from all the partitions is summed together, it still
> doesn't reach the QOS's limit of 3,000 jobs. 
> 
> Isn't there still an accounting issue somewhere?

It's possible, but there may be a better explanation for that elsewhere I haven't found yet.

If the "owner" QOS was being used as not just the partition QOS on all these condo partitions, but the QOS for the job itself, that may explain that odd result.

> Plus, we're regularly adding new partitions, as new groups join our condo
> cluster. So having multiple QOSes with identical limits seems a bit
> redundant.

Yeah - different way of thinking about it. From Slurm's perspective, "PartitionQOS" is all of the QOS capabilities applied to the partition rather than the individual jobs, and that includes the ability to be used across multiple partitions. Although I can see that's not obvious, and should be better documented.

I'll think more on how to do that, and if there may be a way to add some warnings to the log files to hint at the potential complications.

> To take a step back, what we're trying to do is:
> * establish a set of default limits for our regular users, who can submit to
> a restricted set of partitions,
> * override those restrictions with higher default limits for our owners,
> when they submit jobs to their own partitions.
> And all of that without having the users to specify any QOS at submission
> time, nor using a job submission plugin. :)
> 
> Anyway, of specific partition QOSes is the only way, we can live with it.

The recommendation I usually give as part of the training is, if you're going to use Partition QOSs, to set each of the QOS up as part_$partname through sacctmgr. As there's no internal differentiation between the QOS and partition QOSs that helps highlight the difference between them, and in your case I believe all of them would apply to a single partition.

> But it still looks like there's something amiss with the counting of jobs,
> don't you think?

Let's get the QOS cleared up, and see if you notice that again. I haven't seen an easy path into reproducing that behavior, or anything particularly unusual in the log files thus far.

(In reply to Tim Wickberg from comment #29)
> > So even if usage from all the partitions is summed together, it still
> > doesn't reach the QOS's limit of 3,000 jobs. 
> > 
> > Isn't there still an accounting issue somewhere?
> 
> It's possible, but there may be a better explanation for that elsewhere I
> haven't found yet.
> 
> If the "owner" QOS was being used as not just the partition QOS on all these
> condo partitions, but the QOS for the job itself, that may explain that odd
> result.

Jobs were submitted with the following command:
$ srun -J sdev -t 01:00:00 -p akundaje --mem 4GB --pty bash

ie. without explicitly requesting a QOS, so I'm assuming the default QOS (normal) was used:

# sacctmgr show assoc where user=csfoo format=user,qos,defaultqos
      User                  QOS   Def QOS 
---------- -------------------- --------- 
     csfoo          long,normal    normal


> The recommendation I usually give as part of the training is, if you're
> going to use Partition QOSs, to set each of the QOS up as part_$partname
> through sacctmgr. As there's no internal differentiation between the QOS and
> partition QOSs that helps highlight the difference between them, and in your
> case I believe all of them would apply to a single partition.

Gotcha, I'll do that. I was so hoping of getting rid of the many similar QOSes we've been using so far. :)

> Let's get the QOS cleared up, and see if you notice that again. I haven't
> seen an easy path into reproducing that behavior, or anything particularly
> unusual in the log files thus far.

Sounds good, thanks!

Cheers,
-- 
Kilian

(In reply to Kilian Cavalotti from comment #30)
> Gotcha, I'll do that. I was so hoping of getting rid of the many similar
> QOSes we've been using so far. :)
> 
> > Let's get the QOS cleared up, and see if you notice that again. I haven't
> > seen an easy path into reproducing that behavior, or anything particularly
> > unusual in the log files thus far.
> 
> Sounds good, thanks!

All right, I think I'm convinced now. 

I was able to reproduce the problem by submitting jobs to 3 different partitions that use the same Partition QOS (with a MaxJobsPU of 3,000): after submitting 1,000 jobs to each of those 3 partitions, I couldn't submit another job to any of them, and was receiving the "Job violates accounting/QOS policy" message.

So I've implemented the solution you suggested: specific partition QOSes for each partition.

I still feel this could be made more explicit in the documentation, because I don't think I'll the only one to believe that partition QOS limits only apply to a single partition.

Thanks for the clarification and suggestions, as always!

Cheers,
--
Kilian

Hi Tim, 

So, although I still understand that the previous situation of sharing the same partition QOS for multiple partitions was a recipe for limits snafu, there still seems to be something amiss.

We now have partition-specific partition QOSes (quite the mouthful), and still, "scontrol show assoc" reports more the jobs than users are actually submitting.

I'm attaching new "squeue" and "scontrol show assoc" outputs, that shows that user "csfoo" has 74 jobs in queue, while the scheduler thinks he has 98.

Files generated with:
$ squeue -u csfoo -t all > squeue-t11.txt; scontrol show assoc > scontrol-assoc-t11.txt


Any insight?

Cheers,
--
Kilian

Created attachment 4728 [details]
squeue-t11.txt

Created attachment 4729 [details]
scontrol-assoc-t11.txt

Hi SchedMD!

New update about the job count discrepancies between "squeue" and "scontrol show assoc". Here's what our user reported:

-- 8< --------------------------------------------------------------------
Haven't been able to reproduce the exact issue again, but here's a more 
dramatic example of what seems to be an accounting curiosity - I have 9 
jobs in part_kundaje QOS but it thinks I have 353 submitted, and the 
numbers for the GrpSubmitJobs also don't seem to add up.

As usual, outputs of squeue and scontrol are attached, generated with

squeue -u csfoo -t all > squeue-t13.txt; scontrol show assoc > 
scontrol-assoc-t13.txt



QOS=part_akundaje(132) 
UsageRaw=12485377.071585 
GrpJobs=N(33) GrpSubmitJobs=N(205) GrpWall=N(86483.82) 
GrpTRES=cpu=N(113),mem=N(692317),energy=N(0),node=N(33),gres/gpu=N(0)

GrpTRESMins=cpu=N(208089),mem=N(2038367787),energy=N(0),node=N(86483),gres/gpu=N(0)

GrpTRESRunMins=cpu=N(214331),mem=N(2243087158),energy=N(0),node=N(90514),gres/gpu=N(0) 
MaxWallPJ=10080 
MaxTRESPJ= 
MaxTRESPN= 
MaxTRESMinsPJ= 
MinTRESPJ= 
PreemptMode=OFF 
Priority=0 
Account Limits 
Account akundaje 
MaxJobsPA=N(33) MaxSubmitJobsPA=5000(205)

MaxTRESPA=cpu=99999(113),mem=N(692317),energy=N(0),node=N(33),gres/gpu=N(0) 
User Limits 
User 36268 
MaxJobsPU=N(9) 
MaxSubmitJobsPU=3000(353)

MaxTRESPU=cpu=99999(17),mem=N(212317),energy=N(0),node=N(9),gres/gpu=N(0) 
User 312359 
MaxJobsPU=N(24) 
MaxSubmitJobsPU=3000(0)

MaxTRESPU=cpu=99999(96),mem=N(480000),energy=N(0),node=N(24),gres/gpu=N(0)

-- 8< --------------------------------------------------------------------

Created attachment 4747 [details]
squeue-t13.txt

Created attachment 4748 [details]
scontrol-assoc-t13.txt

Hi SchedMD!

Monday update: the issue is still here. 

Despite now having distinct QOSes for each partition, the user still managed to reproduce the issue where, after some time, his job submissions are rejected with "Job violates accounting/QOS policy"

Here are the commands used to generate the "squeue" and "scontrol show assoc" outputs that are gonna be attached:

-- 8< ---------------------------------------------------------------------
$ squeue -u csfoo -t all > squeue-t16.txt; scontrol show assoc > scontrol-assoc-t16.txt 
$ /usr/bin/srun -J sdev -t 24:00:00 -p akundaje --mem 4GB --pty /bin/bash -i -l 
srun: error: Unable to allocate resources: Job violates accounting/QOS 
policy (job submit limit, user's size and/or time limits) 
-- 8< ---------------------------------------------------------------------


User Limits 
User 36268 
MaxJobsPU=N(4) 
MaxSubmitJobsPU=3000(3000)

MaxTRESPU=cpu=99999(10),mem=N(89688),energy=N(0),node=N(4),gres/gpu=N(0)


We would appreciate any update if you got a chance.


Thanks!
--
Kilian

Created attachment 4756 [details]
squeue-t16.txt

Created attachment 4757 [details]
scontrol-assoc-t16.txt

Hi SchedMD!

Just a friendly ping to see if you had any chance to look into this. 

We're still experiencing this issue, and it's starting to be blocking for our users.

Thanks!
-- 
Kilian

Hi!

I'm taking the liberty to bump the Importance of that bug, since over the week-end, several other users reported the issue. It seems to happen on different partitions, using different QOSes, and for different users.

For instance, user "possu(306840)" has 0 (zero) job in queue, and yet, Slurm considers it submitted 2638:

# squeue -u possu
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
# scontrol show assoc
QOS=owners(58)
    UsageRaw=935234674.078245
    GrpJobs=N(9) GrpSubmitJobs=N(2613) GrpWall=N(1656485.69)
    GrpTRES=cpu=N(540),mem=N(3936000),energy=N(0),node=N(29),gres/gpu=N(0)
    GrpTRESMins=cpu=N(15587056),mem=N(92845143570),energy=N(0),node=N(2215394),gres/gpu=N(0)
    GrpTRESRunMins=cpu=N(1168127),mem=N(8639433733),energy=N(0),node=N(64115),gres/gpu=N(0)
    MaxWallPJ=2880
    MaxTRESPJ=
    MaxTRESPN=
    MaxTRESMinsPJ=
    MinTRESPJ=
    PreemptMode=OFF
    Priority=0
[...]
    Account Limits
[...]
      Account possu
        MaxJobsPA=N(0) MaxSubmitJobsPA=5000(2638)
        MaxTRESPA=cpu=4096(0),mem=N(0),energy=N(0),node=N(0),gres/gpu=N(0)
[...]
    User Limits
[...]
      User 306840
        MaxJobsPU=N(0)
        MaxSubmitJobsPU=3000(2638)
        MaxTRESPU=cpu=2048(0),mem=N(0),energy=N(0),node=N(0),gres/gpu=N(0)


I'll attach the full output of "squeue" and "scontrol show assoc"

Please let us know if we can provide any more information to resolve this issue. As I mentioned, this is getting more and more pressing, as it potentially affects every user and prevent them to submit jobs after some time.

Thanks!
-- 
Kilian

Created attachment 4789 [details]
scontrol-assoc.txt

Created attachment 4790 [details]
squeue.txt

Sorry, Killian, I'll see what I can find on this.

Killian, in your last report from comments 43-45 you will find

ClusterName=sherlock Account=possu UserName=parberg(294523) Partition= ID=6940
    SharesRaw/Norm/Level/Factor=100/0.12/800/0.05
    UsageRaw/Norm/Efctv=6866827.41/0.00/0.07
    ParentAccount= Lft=5406 DefAssoc=Yes
    GrpJobs=N(128)
    GrpSubmitJobs=N(600) GrpWall=N(114447.51)
    GrpTRES=cpu=N(128),mem=N(512000),energy=N(0),node=N(128),gres/gpu=N(0)
    GrpTRESMins=cpu=N(114447),mem=N(457788493),energy=N(0),node=N(114447),gres/gpu=N(0)
    GrpTRESRunMins=cpu=N(129246),mem=N(516987600),energy=N(0),node=N(129246),gres/gpu=N(0)
    MaxJobs= MaxSubmitJobs= MaxWallPJ=
    MaxTRESPJ=
    MaxTRESPN=
    MaxTRESMinsPJ=

Is the one using up resources for account possu (not the user possu).

Looking at (what I am guessing is the job's QOS 'normal')
QOS=normal(3)
...
      Account possu
        MaxJobsPA=N(128) MaxSubmitJobsPA=3000(600)
        MaxTRESPA=cpu=256(128),mem=N(512000),energy=N(0),node=N(128),gres/gpu=N(0)


Could you send me the output of

scontrol show job 275581 (or 275582, 275583, 275584)

I am guessing they are asking for more than 128 cpus on those jobs?

Hi Dany, 

Thanks for the feedback, much appreciated.

(In reply to Danny Auble from comment #46)
> Killian, in your last report from comments 43-45 you will find
> 
> ClusterName=sherlock Account=possu UserName=parberg(294523) Partition=
> ID=6940
>     GrpSubmitJobs=N(600) GrpWall=N(114447.51)
>     GrpTRES=cpu=N(128),mem=N(512000),energy=N(0),node=N(128),gres/gpu=N(0)

> Is the one using up resources for account possu (not the user possu).

"parberg" has jobs running indeed, but only 129, not the 2,600+ that are reported by "scontrol show assoc" for user "possu":

# grep -c parberg squeue.txt 
129


> Could you send me the output of
> scontrol show job 275581 (or 275582, 275583, 275584)

Sure (attached in jobs.txt).

> I am guessing they are asking for more than 128 cpus on those jobs?

No, they apparently each request 1 node, 16 CPUs.

And I'm confused: the limit being hit is MaxSubmitJobsPerUser (cf. instrumented logs in https://bugs.schedmd.com/attachment.cgi?id=4690), so the number of CPUs shouldn't be involved, right?

Thanks for looking into it!

Cheers,
-- 
Kilian

Created attachment 4791 [details]
scontrol show job for 275581 275582 275583 275584

(In reply to Kilian Cavalotti from comment #47
> No, they apparently each request 1 node, 16 CPUs.
> 
> And I'm confused: the limit being hit is MaxSubmitJobsPerUser (cf.
> instrumented logs in https://bugs.schedmd.com/attachment.cgi?id=4690), so
> the number of CPUs shouldn't be involved, right?

I was only going on squeue, those jobs list QOSMaxCpuPerUserLimit.  I notice 2 of those jobs I requested information or did start running.  I am guessing these are not related.

I did notice a potential issue though related to what you are seeing.  I will attach a patch in a moment.

You wouldn't happen to know if csfoo is submitting array's that are larger than 3000?

Created attachment 4792 [details]
Patch to correctly remove jobs from qos submitted jobs

Killian, please try this patch and report back if things correct themselves.

Before trying this patch (and assuming the counts are still in question) please restart the slurmctld and see if it fixes things by just restarting.

(In reply to Danny Auble from comment #49)
> I was only going on squeue, those jobs list QOSMaxCpuPerUserLimit.  I notice
> 2 of those jobs I requested information or did start running.  I am guessing
> these are not related.

Oh yeah, ok. We indeed have cpu=128 MaxTRESPU in the "normal" QOS. 

The blocking we're seeing is about MaxSubmitPU in the "owners" and "part_<owner>" partition QOSes. (I know, the ticket history is a bit long, sorry...)

> I did notice a potential issue though related to what you are seeing.  I
> will attach a patch in a moment.

Great!

> You wouldn't happen to know if csfoo is submitting array's that are larger
> than 3000?

I know he's not, because:

# scontrol show config | grep MaxArray
MaxArraySize            = 1001

Thanks!
-- 
Kilian

(In reply to Danny Auble from comment #50)
> Created attachment 4792 [details]
> Patch to correctly remove jobs from qos submitted jobs
> 
> Killian, please try this patch and report back if things correct themselves.

Will do, thanks.

> Before trying this patch (and assuming the counts are still in question)
> please restart the slurmctld and see if it fixes things by just restarting.

Yes, restarting slurmctld has been confirmed to clear up the "ghost jobs" counts and unblock the situation for a time, until it happens again.

Will try the patch right now.

Cheers,
-- 
Kilian

Awesome.  This code was definitely wrong before, as it is directly related to the problem you are seeing I am hoping this will fix the situation :).

(In reply to Danny Auble from comment #53)
> Awesome.  This code was definitely wrong before, as it is directly related
> to the problem you are seeing I am hoping this will fix the situation :).

Great, me too!

Patch is deployed, I will get back in touch with the users to ask them to submit jobs and will report back.

Thanks!
-- 
Kilian

Killian, I just reproduced the issue.  It has to deal with job's moving partitions while pending.  It doesn't look like the accounting was thought out very well for this situation.

I'll see if I can get a patch that will handle at least the simple scenarios.  I'll let you know what I come up with.

I will make note the last patch (now commit a034e4ab8080) is an improvement to the code, but will not solve the scenario.

(In reply to Danny Auble from comment #55)
> Killian, I just reproduced the issue.  It has to deal with job's moving
> partitions while pending.  It doesn't look like the accounting was thought
> out very well for this situation.
> 
> I'll see if I can get a patch that will handle at least the simple
> scenarios.  I'll let you know what I come up with.
> 
> I will make note the last patch (now commit a034e4ab8080) is an improvement
> to the code, but will not solve the scenario.

Aaah, so that would be the case for jobs submitted to multiple partitions, right? Our users are especially fond of those...

Cheers,
-- 
Kilian

(In reply to Kilian Cavalotti from comment #56)
> Aaah, so that would be the case for jobs submitted to multiple partitions,
> right? Our users are especially fond of those...

And I just confirmed that all the users affected by this were indeed submitting jobs to multiple partitions. Good catch!

Cheers,
-- 
Kilian

You got it.  Or if someone changed the partition with scontrol.

Glad you were able to confirm :).

One note, behavior will change here on job submit.  If the user breaks any partition's QOS MaxSubmit* limit they will be deigned from running.

Also note if you have the same QOS as the partition QOS the submit count will go up multiple times.

For instance

PartitionName=part1 qos=part_qos
PartitionName=part2 qos=part_qos

sbatch -ppart1,part2

Then the count for part_qos would be MaxSubmitJobsPA=N(2), making it possible to break a limit of 10 with 1 job if you had the same qos in 11 different partitions.

Does that make sense?  I don't know if you do it this way (not advised) but I have seen some people do do it this way ;).

Created attachment 4793 [details]
Patch to handle multiple partitions that have a partition QOS for submit limits

Killian, I believe this patch will fix the scenario.  I have tested if fairly well, but when you use it I would monitor the output of scontrol show assoc every once in a while to make sure nothing looks totally out of place.

What this patch does is different than I explained before in respects to a job counting multiple times if the partition qos was the same for multiple partitions.  It does not do that, it only counts it once no matter how many partitions the job is submitted to.

In any case.  Please let me know how this looks on your system, if you see anything out of the ordinary or not.  We will try to get this into 17.02.5 (due this week).

Thanks for your patience on the matter.

Created attachment 4794 [details]
Patch to handle multiple partitions that have a partition QOS for submit limits

We just found a bad line (shouldn't cause any real issue).  I just fixed it in this patch.

List part_qos_list = part_qos_list = list_create(NULL);

No reason to do that double set huh ;).

(In reply to Danny Auble from comment #61)
> Created attachment 4794 [details]
> Patch to handle multiple partitions that have a partition QOS for submit
> limits
> 
> We just found a bad line (shouldn't cause any real issue).  I just fixed it
> in this patch.
> 
> List part_qos_list = part_qos_list = list_create(NULL);
> 
> No reason to do that double set huh ;).

Awesome, thanks Danny!
Patch #2 applied and deployed, will monitor things and report back.

Cheers,
-- 
Kilian

Just a quick update to say that things look good so far: "scontrol show assoc" has been consistently accurate in reporting the number of jobs in queue for each user.

So, I'll wait a few more days, but that looks very promising, thanks!

Cheers,
-- 
Kilian

Fantasic, we did find one minor issue that it doesn't appear your environment has.  It has to deal with no partitions the job was submitted to not having any qos.  If all your partitions have a QOS then the difference wouldn't matter.

The final patch is in commit 4b003d37ab5 and will be in 17.02.5 most likely tagged later this week.

Kilian, please reopen this if you still experience issues.  I think this one is solved though :).

(In reply to Danny Auble from comment #70)
> Kilian, please reopen this if you still experience issues.  I think this one
> is solved though :).

Great job in fixing this, thank you!

Cheers,
-- 
Kilian

Hello.   We would like this fix in the next version of 16.05

(https://github.com/SchedMD/slurm/commit/4b003d37ab5)

Thanks.

I am sorry, there is no current plan to release another 16.05.  I would suggest locally backporting the patch until you are able to upgrade to 17.02+.

Thanks!

No bother :).

Unfortunately this will not be backported.  You will still have to locally backporting the patch until you are able to upgrade to 17.02+.

Note, support for 16.05 will end in ~2 months.

Thanks!