Ticket 2423

Hey Michael, I'll look into what this would take.  I'll get something back to you in a bit.  It will probably be a configuration option in the slurmdbd.conf.

The development will probably only support the current usagefactor of the QOS, so if the QOS usagefactor changes over time it will only use the latest value.

Would you want this to work for sacct as well as sreport, or just sreport?  I'm guessing both.

Our primary concern is that users submitting jobs to a qos with usageFactor=0 do not end up pending with (AssocGrpCPUMinutesLimit) 
Secondarily jobs run in the scavenger qos do not get "billed" for usage
ie The amount that an account is "billed" for usage is base multiplied by the 
usageFactor
Finally, As far as reporting is concerned, we would need any one tool(probably sreport) that would allow us to get total utilization and billed utilization per user and account. We need to report in near real time "remaining hours" to users and quarterly to allocation provisioners to inform reallocation of resources.   

Current usageFactor should be fine as the usage factors do not change frequently(if ever) 
Our current reporting tools are based on sreport cluster AccountUtilizationByUser  and scontrol show cache so we do not have a priority for sacct

Ah, I see, the usageFactor currently does get applied to all the time based limits on the association as well as QOS.  It just doesn't get applied to the numbers sacct and sreport report.

Are you seeing something else as far as this is concerned, or are you just reporting what you understood the docs to say?

I just checked and this is how the UsageFactor option reads...

UsageFactor Usage factor when running with this QOS (i.e. .5 would make it use only half the time as normal in accounting and 2 would make it use twice as much.) This only applies to the usage used in determining priority and doesn't modify the real usage in the database. 

In this case the usage used to determine priority is the same usage applied to limits as well.  If you have suggestions how to make that clearer please let me know your ideas :).

Are you still interested in at least sreport reporting the time with this usage factor applied?  Keep in mind you would easily be able to report more time than is possible on your cluster if the usage factor is above 1.

Below is what I am experiencing.
TLDR jobs submitted to a qos with usagefactor=0 are not running

[root@mgmt1 ~]# sacctmgr show associations -o format=account,grptresmin 
   Account   GrpTRESMins
---------- -------------
   verhaag        cpu=24
[root@mgmt1 ~]# sacctmgr show qos -o format=name,usagefactor
      Name UsageFactor
---------- -----------
    normal    1.000000
 scavenger    0.000000

[mverhaa1@jhu.edu@mgmt1 ~]$ cat sleepy.sh
#!/bin/bash

#SBATCH
#SBATCH --qos=scavenger
#SBATCH --time=00:01:00
#SBATCH --nodes=1
#SBATCH --partition=scavenger
#SBATCH --account=verhaag
#SBATCH --cpus-per-task=24
sleep 60
[mverhaa1@jhu.edu@mgmt1 ~]$  scontrol show cache |grep verhaag -A 13 |tail -n 26
    SharesRaw/Norm/Level/Factor=1/0.04/2/0.00
    UsageRaw/Norm/Efctv=0.00/0.00/0.00
    ParentAccount=marcc(5) Lft-Rgt=87-80 DefAssoc=No
    GrpJobs=N(0)
    GrpSubmitJobs=N(0) GrpWall=N(0.00)
    GrpTRES=cpu=N(0),mem=N(0),energy=N(0),node=N(0),gres/gpu=N(0)
    GrpTRESMins=cpu=24(0),mem=N(0),energy=N(0),node=N(0),gres/gpu=N(0)
    GrpTRESRunMins=cpu=N(0),mem=N(0),energy=N(0),node=N(0),gres/gpu=N(0)
    MaxJobs= MaxSubmitJobs= MaxWallPJ=
    MaxTRESPJ=cpu=3600
    MaxTRESPN=
    MaxTRESMinsPJ=
ClusterName=marcc Account=verhaag UserName=mverhaa1@jhu.edu(1058) Partition= ID=860
    SharesRaw/Norm/Level/Factor=1/0.04/1/0.00
    UsageRaw/Norm/Efctv=0.00/0.00/4294967294.00
    ParentAccount= Lft-Rgt=88-81 DefAssoc=Yes
    GrpJobs=N(0)
    GrpSubmitJobs=N(0) GrpWall=N(0.00)
    GrpTRES=cpu=N(0),mem=N(0),energy=N(0),node=N(0),gres/gpu=N(0)
    GrpTRESMins=cpu=N(0),mem=N(0),energy=N(0),node=N(0),gres/gpu=N(0)
    GrpTRESRunMins=cpu=N(0),mem=N(0),energy=N(0),node=N(0),gres/gpu=N(0)
    MaxJobs= MaxSubmitJobs= MaxWallPJ=
    MaxTRESPJ=cpu=3600
    MaxTRESPN=
    MaxTRESMinsPJ=
[mverhaa1@jhu.edu@mgmt1 ~]$ squeue |grep mverhaa1
[mverhaa1@jhu.edu@mgmt1 ~]$ sbatch sleepy.sh
Submitted batch job 3907033
[mverhaa1@jhu.edu@mgmt1 ~]$  scontrol show cache |grep verhaag -A 13 |tail -n 26
    SharesRaw/Norm/Level/Factor=1/0.04/2/0.00
    UsageRaw/Norm/Efctv=0.00/0.00/0.00
    ParentAccount=marcc(5) Lft-Rgt=87-80 DefAssoc=No
    GrpJobs=N(0)
    GrpSubmitJobs=N(0) GrpWall=N(0.00)
    GrpTRES=cpu=N(0),mem=N(0),energy=N(0),node=N(0),gres/gpu=N(0)
    GrpTRESMins=cpu=24(24),mem=N(132258),energy=N(0),node=N(1),gres/gpu=N(0)
    GrpTRESRunMins=cpu=N(0),mem=N(0),energy=N(0),node=N(0),gres/gpu=N(0)
    MaxJobs= MaxSubmitJobs= MaxWallPJ=
    MaxTRESPJ=cpu=3600
    MaxTRESPN=
    MaxTRESMinsPJ=
ClusterName=marcc Account=verhaag UserName=mverhaa1@jhu.edu(1058) Partition= ID=860
    SharesRaw/Norm/Level/Factor=1/0.04/1/0.00
    UsageRaw/Norm/Efctv=0.00/0.00/4294967294.00
    ParentAccount= Lft-Rgt=88-81 DefAssoc=Yes
    GrpJobs=N(0)
    GrpSubmitJobs=N(0) GrpWall=N(0.00)
    GrpTRES=cpu=N(0),mem=N(0),energy=N(0),node=N(0),gres/gpu=N(0)
    GrpTRESMins=cpu=N(24),mem=N(132258),energy=N(0),node=N(1),gres/gpu=N(0)
    GrpTRESRunMins=cpu=N(0),mem=N(0),energy=N(0),node=N(0),gres/gpu=N(0)
    MaxJobs= MaxSubmitJobs= MaxWallPJ=
    MaxTRESPJ=cpu=3600
    MaxTRESPN=
    MaxTRESMinsPJ=
[mverhaa1@jhu.edu@mgmt1 ~]$ sbatch sleepy.sh
Submitted batch job 3907034
[mverhaa1@jhu.edu@mgmt1 ~]$ squeue |grep mverhaa1
           3907034 scavenger sleepy.s mverhaa1 PD       0:00      1 (AssocGrpCPUMinutesLimit)

Michael, I can confirm what you see.  I can also see in the code where it isn't happening.  Let me see what I can do for you on the primary concern.

Michael, indeed this is working as documented/currently intended.  I do agree with you this is a good idea though.  I'll work on what needs to be done with this and get your a SOW to look at.  Most of it seems straight forward though.  This wouldn't be put into 15.08 either, but most likely in 16.05.

I'd like to add that we are also affected by this problem- I'd originally submitted a patch for this in Bug 1048.
I've updated the patch (attached here) to work with 15.08.8 which should resolve this for us.

Kevin

Created attachment 2821 [details]
Patch to allow job submission limits to reflect usage_factor

Thanks Kevin, I'll take a look at this later and see if we can get it in before 16.05.