Hi David,

I think that the confusion is the following one: when a users/accounts are granted to a reservation and that reservation has some time not being used, that unused time is equally split and "charged" to all users/accounts granted to it in sreport, but not in sshare.

More in general, the confusion is between the different meanings of "resource usage" in slurmctld and slurmdbd.
The value in slurmctld is more a weight related to "usage" but also impacted by things like PriorityDecayHalfLife or UsageFactor.
The value in slurmdbd are actual "hours", including reservations.

Does it make sense to you too?

Regards,
Albert

Hi Albert,

I have read your response a few times, but it still isn't clicking for me completely. I do understand that you're highlighting a difference in functionality and nomenclature.

I think, in part, I am getting caught up on the words "reservation" for sreport and sshare as I tend to think of reservations as a specific, and perhaps narrow, type of thing.

Might there be a spot in the documentation that could help elucidate these distinctions?

David

Hi David,

> I have read your response a few times, but it still isn't clicking for me
> completely. I do understand that you're highlighting a difference in
> functionality and nomenclature.

Yes, it's not easy to explain neither.. I'll try my best! ;-)

> I think, in part, I am getting caught up on the words "reservation" for
> sreport and sshare as I tend to think of reservations as a specific, and
> perhaps narrow, type of thing.

Well, actually the confusion with the Reseration word was not tied to sreport and sshare.
The main confusion was with the "Reserved" word in sreport.
In version 21.08 we renamed that to "Planned" to avoid confusions:

- https://github.com/SchedMD/slurm/blob/slurm-21-08-0-1/RELEASE_NOTES#L56


Anyway, the key word in your case that I wanted to highlight was not "Reservation", but "Usage".
I'll elaborate the 4 main definitions of "usage" in our context:

A) Actual CPU/Resources time used by the actual linux process
  - A job allocating 2 CPUs and running 1 sleep for 30min will have ~0s of "usage".
  - This is gathered by JobAcctGatherType and seen in sacct in UserCPU, SystemCPU and TotalCPU.
  - It's a bit irrelevant from the point of view of Slurm (cannot control it), but not for HPC in general.

B) The CPU/Resources time that Slurm allowed a job to run its processes
  - A job allocating 2 CPUs and running 1 sleep for 30min will have 60min of "usage".
  - This is main values used for Slurm, and can be seen for example in sacct in AllocTRES.
  - The seff tool is aiming to see the difference/ration between A and B.

C) The "scheduler bill" charged to a User/Account due a job
  - This may be equal to B, but things like Usagefactor may increase or decrease it.
  - It's the actual "usage" value managed internally for Slurm for priorities in things like FairShare (sshare),
    and it may decay.

D) The "cluster bill" charged to a user/account in a period of time (sreport)
  - This may also be like B, but if a user/account was granted into a Reservation that was not fully used,
    then they are also charged for that idle time of the Reservation.
    Note that in a way "a Reservation" is also "an Allocation", meaning that some resources are allocated/reserved,
    but instead of directly to a specific single job, to some users/accounts to submit jobs. Therefore it makes sense
    to charge them also for the idle time that they wasted.
  - This is what sreport shows as "Used/Allocated": the time "used in jobs" plus the time "granted but unused in a reservations".

In your case you are kind of expecting C and D to be the same values.
And their values might be equal in some scenarios, but not always.
Actually all 4 "usages" might be the same value in some cases.
And probably the B is the most common interpretation.

> Might there be a spot in the documentation that could help elucidate these
> distinctions?

Not really about the differences above.
But in your case, as you don't use PriorityDecayHalfLife and neither UsageFactor (I think), then the source of differences between C and D is the Reservations granted to a user/account and how sreport charge them.
And here you can read some notes about it:

- https://slurm.schedmd.com/sreport.html#OPT_cluster-AccountUtilizationByUser
- https://slurm.schedmd.com/sreport.html#OPT_cluster-Utilization

Hope it helps,
Albert

Hi David,

If this is ok for you I'm closing this ticket as infogiven.
Please don't hesitate if you need further support.

Regards,
Albert

Hi Albert,

Thanks for this, and apologies for re-opening. Your outline was helpful and filled in some gaps in understanding I had, and shed some light on the moving parts. I did have a couple of questions.


> C) The "scheduler bill" charged to a User/Account due a job
>   - This may be equal to B, but things like Usagefactor may increase or
> decrease it.
>   - It's the actual "usage" value managed internally for Slurm for
> priorities in things like FairShare (sshare),
>     and it may decay.

You say that "it may decay". Is this true even if we do NOT have/use PriorityDecayHalfLife? This is the use case that sparked this ticket on our end, so I want to be sure I'm understanding.


> D) The "cluster bill" charged to a user/account in a period of time (sreport)
>   - This may also be like B, but if a user/account was granted into a
> Reservation that was not fully used,
>     then they are also charged for that idle time of the Reservation.
>     Note that in a way "a Reservation" is also "an Allocation", meaning that
> some resources are allocated/reserved,
>     but instead of directly to a specific single job, to some users/accounts
> to submit jobs. Therefore it makes sense
>     to charge them also for the idle time that they wasted.
>   - This is what sreport shows as "Used/Allocated": the time "used in jobs"
> plus the time "granted but unused in a reservations".

This is helpful because it sheds light on why we are asked if we had reservations when investigating billing anomalies. I may have asked this in the past (in another ticket) but is it safe to think of D as the sum total/aggregate of jobs reported in B, without the reservation factor of course?


> In your case you are kind of expecting C and D to be the same values.
> And their values might be equal in some scenarios, but not always.
> Actually all 4 "usages" might be the same value in some cases.
> And probably the B is the most common interpretation.

This helps to know that they *can* align, but it's not unusual to see them otherwise.

> Not really about the differences above.
> But in your case, as you don't use PriorityDecayHalfLife and neither
> UsageFactor (I think), then the source of differences between C and D is the
> Reservations granted to a user/account and how sreport charge them.
> And here you can read some notes about it:

This is correct. I do not recall a formal setting for "UsageFactor", I apologize. Is there something else that is meant by that? I did a quick run through our conf files, the man pages, etc.


David

I submitted too soon. I found UsageFactor stuff in the QOS documentation. We are doing nothing profound with usage factor from what I can see:


```
[drhey@glctld slurm-21.08.8-2]$ date; sacctmgr show qos format=cluster,name,usagefactor -p
Mon Nov 28 11:07:22 EST 2022
Cluster|Name|UsageFactor|
|normal|1.000000|
|interactive|1.000000|
|preempt|1.000000|
|largemem|1.000000|
|expedite|1.000000|
|training|1.000000|
|gpu|1.000000|
|debug|1.000000|
|standard-oc|1.000000|
|class|1.000000|
|team|1.000000|
|admin|0.000000|
|spgpu|1.000000|
```

Hi David,

> Thanks for this, and apologies for re-opening.

No apologies, that's my job! ;-)

> Your outline was helpful and
> filled in some gaps in understanding I had, and shed some light on the
> moving parts.

Great!

> I did have a couple of questions.
> 
> 
> > C) The "scheduler bill" charged to a User/Account due a job
> >   - This may be equal to B, but things like Usagefactor may increase or
> > decrease it.
> >   - It's the actual "usage" value managed internally for Slurm for
> > priorities in things like FairShare (sshare),
> >     and it may decay.
> 
> You say that "it may decay". Is this true even if we do NOT have/use
> PriorityDecayHalfLife? This is the use case that sparked this ticket on our
> end, so I want to be sure I'm understanding.

No, the "may" means "depending on options like PriorityDecayHalfLife or PriorityUsageResetPeriod, it may decay".
In your case it shouldn't decay.

> > D) The "cluster bill" charged to a user/account in a period of time (sreport)
> >   - This may also be like B, but if a user/account was granted into a
> > Reservation that was not fully used,
> >     then they are also charged for that idle time of the Reservation.
> >     Note that in a way "a Reservation" is also "an Allocation", meaning that
> > some resources are allocated/reserved,
> >     but instead of directly to a specific single job, to some users/accounts
> > to submit jobs. Therefore it makes sense
> >     to charge them also for the idle time that they wasted.
> >   - This is what sreport shows as "Used/Allocated": the time "used in jobs"
> > plus the time "granted but unused in a reservations".
> 
> This is helpful because it sheds light on why we are asked if we had
> reservations when investigating billing anomalies. I may have asked this in
> the past (in another ticket) but is it safe to think of D as the sum
> total/aggregate of jobs reported in B, without the reservation factor of
> course?

Yes.

> > In your case you are kind of expecting C and D to be the same values.
> > And their values might be equal in some scenarios, but not always.
> > Actually all 4 "usages" might be the same value in some cases.
> > And probably the B is the most common interpretation.
> 
> This helps to know that they *can* align, but it's not unusual to see them
> otherwise.

Exaclty.
And they are all "usage", but just slightly different.
So, I totally get the "confusion".

> > Not really about the differences above.
> > But in your case, as you don't use PriorityDecayHalfLife and neither
> > UsageFactor (I think), then the source of differences between C and D is the
> > Reservations granted to a user/account and how sreport charge them.
> > And here you can read some notes about it:
> 
> This is correct. I do not recall a formal setting for "UsageFactor", I
> apologize. Is there something else that is meant by that? I did a quick run
> through our conf files, the man pages, etc.

Actually, UsageFactor is not in the config files but in the DB as a QOS parameter.
And as mentioned above, the PriorityUsageResetPeriod will also alter C).

Right now I think that no other parameter will change C (besides PriorityDecayHalfLife, PriorityUsageResetPeriod and UsageFactor), but please note that there is no garantee that new Slurm parameters/features will also have some internal impact in C.

Regards,
Albert

Albert,

> > > C) The "scheduler bill" charged to a User/Account due a job
> > >   - This may be equal to B, but things like Usagefactor may increase or
> > > decrease it.
> > >   - It's the actual "usage" value managed internally for Slurm for
> > > priorities in things like FairShare (sshare),
> > >     and it may decay.
> > 
> > You say that "it may decay". Is this true even if we do NOT have/use
> > PriorityDecayHalfLife? This is the use case that sparked this ticket on our
> > end, so I want to be sure I'm understanding.
> 
> No, the "may" means "depending on options like PriorityDecayHalfLife or
> PriorityUsageResetPeriod, it may decay".
> In your case it shouldn't decay.


Given that we do NOT use PriorityDecayHalfLife or PriorityUsageResetPeriod, is it safe to believe that GrpTRESRaw in the sshare output would never be 0 or blank then? Unless, of course, an account was never used.

David

David,

> > > > C) The "scheduler bill" charged to a User/Account due a job
> > > >   - This may be equal to B, but things like Usagefactor may increase or
> > > > decrease it.
> > > >   - It's the actual "usage" value managed internally for Slurm for
> > > > priorities in things like FairShare (sshare),
> > > >     and it may decay.
> > > 
> > > You say that "it may decay". Is this true even if we do NOT have/use
> > > PriorityDecayHalfLife? This is the use case that sparked this ticket on our
> > > end, so I want to be sure I'm understanding.
> > 
> > No, the "may" means "depending on options like PriorityDecayHalfLife or
> > PriorityUsageResetPeriod, it may decay".
> > In your case it shouldn't decay.
> 
> 
> Given that we do NOT use PriorityDecayHalfLife or PriorityUsageResetPeriod,
> is it safe to believe that GrpTRESRaw in the sshare output would never be 0
> or blank then? Unless, of course, an account was never used.

Yes.
Well, I would say that using a QOS with UsageFactor=0 could also lead to GrpTRESRaw=0 even if some job was run.
But I would say that there is no other case.

I guess that this links back to your initial question, right?
There you were comparing sshare with sreport, and that was overlooking the idle time of reservations that is accounted in sreport.
With your config and assuming that UsageFactor!=0 (and that you never used PriorityDecayHalfLife and PriorityUsageResetPeriod), if GrpTRESRaw is 0 the account shouldn't have any job run neither.

You should be able to double check that with sacct instead of sreport.
Does it match?

Regards,
Albert

Albert,

> Yes.
> Well, I would say that using a QOS with UsageFactor=0 could also lead to
> GrpTRESRaw=0 even if some job was run.
> But I would say that there is no other case.

Thanks. So, a few replies back I showed the output of our QOS' presently. None have a UsageFactor of 0 that a user would use. I have looked and the jobs for this particular account have never used anything other than normal QOS for their jobs in the entire time the cluster has been used:

```
[root@gl-build ~]# date; sacct -XP -A henryliu1 --start=2020-01-06T00:00:00 --end=2022-11-07T00:00:00 --format=User,JobId,Partition,QOS,ExitCode 2>&1 | tee henryliu1.out
Wed Nov 30 08:54:46 EST 2022
...

[root@gl-build ~]# date; awk -F"|" '{print $4}' henryliu1.out | sort | uniq
Wed Nov 30 08:56:34 EST 2022
normal
QOS
```

> I guess that this links back to your initial question, right?
> There you were comparing sshare with sreport, and that was overlooking the
> idle time of reservations that is accounted in sreport.
> With your config and assuming that UsageFactor!=0 (and that you never used
> PriorityDecayHalfLife and PriorityUsageResetPeriod), if GrpTRESRaw is 0 the
> account shouldn't have any job run neither.

Yes, this ties back to the initial question as to why GrpTRESRaw would have a billing=0 when we know there is usage that shows in sreport.

> You should be able to double check that with sacct instead of sreport.
> Does it match?

So, I have an idea of what is meant here and have done something similar in a script in the past. I will get the data from that soon. I've been pulled in a couple directions this morning. 


David

Hi David,

> > Well, I would say that using a QOS with UsageFactor=0 could also lead to
> > GrpTRESRaw=0 even if some job was run.
> > But I would say that there is no other case.

Actually I've just thought that some values of TRESBillingWeights could also make that happen. But it would be rare and not your case.

> So, a few replies back I showed the output of our QOS' presently.
> None have a UsageFactor of 0 that a user would use. I have looked and the
> jobs for this particular account have never used anything other than normal
> QOS for their jobs in the entire time the cluster has been used:

Yes, I agree that in your case the UsageFactor of a QoS doesn't seem like the cause of the difference.

> Yes, this ties back to the initial question as to why GrpTRESRaw would have
> a billing=0 when we know there is usage that shows in sreport.

Please note that as commented in comment 3, the sreport usage includes also reservations.

> > You should be able to double check that with sacct instead of sreport.
> > Does it match?
> 
> So, I have an idea of what is meant here and have done something similar in
> a script in the past. I will get the data from that soon. I've been pulled
> in a couple directions this morning.

Ok, let me know what sacct tell us.


Another option I was wondering is that maybe at some point you run a command like:

$ sacctmgr modify account henryliu1t set RawUsage=0

This command will set the slurmctld usage (C) to 0, while the usage reported by sreport (D) will remain unchanged.
Is that possible?

Regards,
Albert

Hi Albert,

> > > You should be able to double check that with sacct instead of sreport.
> > > Does it match?
> > 
> > So, I have an idea of what is meant here and have done something similar in
> > a script in the past. I will get the data from that soon. I've been pulled
> > in a couple directions this morning.
> 
> Ok, let me know what sacct tell us.

For the henryliu1 account I was able to get the following:

total of all jobs in sacct that ran from the start of our cluster to 2022-12-01: 962894295463.6836
total of all jobs according to sreport from the same start and end point: 962894295464
with the delta between the two being: -0.31640625

But sshare doesn't show any usage at all presently

> Another option I was wondering is that maybe at some point you run a command
> like:
> 
> $ sacctmgr modify account henryliu1t set RawUsage=0
> 
> This command will set the slurmctld usage (C) to 0, while the usage reported
> by sreport (D) will remain unchanged.
> Is that possible?

I plan on talking with my team on this, as it was suggested in bug 13400 too. We would most likely want to have some remediation in place, or some way to note the value before it's set back to 0. I am assuming that because we'd be using sacctmgr it would be targeting the usage within slurmdbd, and not the controller only?

David

Hi David,

> > Ok, let me know what sacct tell us.
> 
> For the henryliu1 account I was able to get the following:
> 
> total of all jobs in sacct that ran from the start of our cluster to
> 2022-12-01: 962894295463.6836
> total of all jobs according to sreport from the same start and end point:
> 962894295464
> with the delta between the two being: -0.31640625

I assume that those are billing values and not really "number of jobs".
And I guess that the delta is just some rounding being done somewhere.
This seems fine, right?

> But sshare doesn't show any usage at all presently

Ok, why that is 0 is the current remaining mistery.

> > Another option I was wondering is that maybe at some point you run a command
> > like:
> > 
> > $ sacctmgr modify account henryliu1t set RawUsage=0
> > 
> > This command will set the slurmctld usage (C) to 0, while the usage reported
> > by sreport (D) will remain unchanged.
> > Is that possible?
> 
> I plan on talking with my team on this, as it was suggested in bug 13400
> too.

Yes, my current guess is that the usage in slurmctld was reseted somehow at some point.

Actually, I'm thinking that another way that you could reset it unintentionally is by removing and recreating the account.
Because from the point of view of the controller it will be a fresh new account (usage C would be 0), but for sacct, if its name is the same, the queries will return the old values too.

I'm just trying to think options that lead you to the current situation...
But I'm glad that at least now we agree that Slurm is not violating a limit.

> We would most likely want to have some remediation in place, or some
> way to note the value before it's set back to 0.

Do you mean that you want to find a way to set the usage in sshare to a higher number?
That's not possible, sorry.
I think that you'll need to work with all the information to set the limit acordingly.
That is, with the sreport/sacct info, plus the sshare/scontrol info, decide which one you trust and set the limit always knowning that it will be compared to the sshare/scontrol value.

An alternative would be actually to follow the Slurm way of thinking on usage and limits.
That is, instead of (manually) changing/increasing limits, let Slurm to (automatically) change/reduce usage (C, not D).
I think that you discarded this in some other issue, but now that you know better about C and D, maybe that makes better sense now.
Maybe you don't want that automatic part, but you could still reset usages to 0 instead of icreasing limits.

I totally understand your thinking on increasing limits, seems natural, but note that in the long-long-long term increasing limits is problably not technically possible, right?
That's also why Slurm uses the Decay usage idea.
So, you set a limit and you know that for a Decay/Reset period of time it won't be violated, but after that period the old usage is not taken into account. So, technically is like the limit has been increased.

> I am assuming that because
> we'd be using sacctmgr it would be targeting the usage within slurmdbd, and
> not the controller only?

Sorry, I'm not certain what exactly do you mean here.
But I would say this:
- The "sacctmgr modify account henryliu1t set RawUsage=0" will reset the usage of sshare/scontrol value (C), not the sacct/sreport. You cannot rewrite History.
- The "sacctmgr modify account henryliu1t set WhateverLimit=xxxx" will modify the limit everywhere.

Regards,
Albert

PS: I wonder if we should close this ticked and focus the discussion in bug 13400, at this point I think they are the same issue/discussion, right?

Hey, Albert!

> I assume that those are billing values and not really "number of jobs".
> And I guess that the delta is just some rounding being done somewhere.
> This seems fine, right?

Correct, that's the billing values. Unfortunately, I can't confirm all time number
of jobs with sreport as it's giving me:

sreport: error: slurm_persist_conn_open_without_init: failed to open persistent connection to host:slurmdbd:6819: Connection refused
sreport: error: Getting response to message type: DBD_GET_JOBS_COND
sreport: error: DBD_GET_JOBS_COND failure: Unspecified error
 Problem with job query.

and the query is sufficiently large (e.g. sreport -nP job SizesByAccount -M greatlakes Start=2020-01-06T12:00:00 End=2022-12-01T00:00:00 PrintJobCount accounts=henryliu1)

The fact that sacct and sreport seem to more or less concur, is good for us. So we can just let this be for now.

> Yes, my current guess is that the usage in slurmctld was reseted somehow at
> some point.
> 
> Actually, I'm thinking that another way that you could reset it
> unintentionally is by removing and recreating the account.
> Because from the point of view of the controller it will be a fresh new
> account (usage C would be 0), but for sacct, if its name is the same, the
> queries will return the old values too.
> 
> I'm just trying to think options that lead you to the current situation...
> But I'm glad that at least now we agree that Slurm is not violating a limit.

I have looked at sacctmgr show transactions for the account and don't see anything pertaining to account removal, 
or a reset raw usage change. Is there any way to determine whether or not such an action occurred by some other
method?


> > We would most likely want to have some remediation in place, or some
> > way to note the value before it's set back to 0.
> 
> Do you mean that you want to find a way to set the usage in sshare to a
> higher number?
> That's not possible, sorry.
> I think that you'll need to work with all the information to set the limit
> acordingly.
> That is, with the sreport/sacct info, plus the sshare/scontrol info, decide
> which one you trust and set the limit always knowning that it will be
> compared to the sshare/scontrol value.

I think what I meant here was that: if we can't find the reason why it's 0 could
we set it again. I now know that we can't, but as you mentioned we still have a 
mystery to solve because it shouldn't be 0 in our case (unless it was somehow
deleted or had the usage set to 0; which I don't know if we can really query
that at this point).

> An alternative would be actually to follow the Slurm way of thinking on
> usage and limits.
> That is, instead of (manually) changing/increasing limits, let Slurm to
> (automatically) change/reduce usage (C, not D).
> I think that you discarded this in some other issue, but now that you know
> better about C and D, maybe that makes better sense now.
> Maybe you don't want that automatic part, but you could still reset usages
> to 0 instead of icreasing limits.
> 
> I totally understand your thinking on increasing limits, seems natural, but
> note that in the long-long-long term increasing limits is problably not
> technically possible, right?
> That's also why Slurm uses the Decay usage idea.
> So, you set a limit and you know that for a Decay/Reset period of time it
> won't be violated, but after that period the old usage is not taken into
> account. So, technically is like the limit has been increased.

This might be something that we could discuss internally; or at least a way
to do this without increasing the limit ad-infinitum
 
> > I am assuming that because
> > we'd be using sacctmgr it would be targeting the usage within slurmdbd, and
> > not the controller only?
> 
> Sorry, I'm not certain what exactly do you mean here.
> But I would say this:
> - The "sacctmgr modify account henryliu1t set RawUsage=0" will reset the
> usage of sshare/scontrol value (C), not the sacct/sreport. You cannot
> rewrite History.
> - The "sacctmgr modify account henryliu1t set WhateverLimit=xxxx" will
> modify the limit everywhere.

I think given the scenarios you outlined (A-D) we can consider this stanza solved, 
because I now understand that the values are independent for some time and sync up
only at starts|restarts of the daemons, and that sshare/slurmctld are the
better source of record for usages.

> PS: I wonder if we should close this ticked and focus the discussion in bug
> 13400, at this point I think they are the same issue/discussion, right?

I am not sure on that. I think we can agree that we are in sync on what information
comes from where, and what's reliable. I am still curious as to how the sshare
GrpTRESRaw can show a billing=0; and if there's any way for us to ascertain the
cause (e.g. reset raw was run, or the account was deleted). I've done the things
I can think of, which are probably limited. If you see them as being similar enough,
I would be OK with continuing the discussion in bug 13400.

David

Hi David,

> The fact that sacct and sreport seem to more or less concur, is good for us.
> So we can just let this be for now.

Good!

> I have looked at sacctmgr show transactions for the account and don't see
> anything pertaining to account removal,

If that happen, it should appear as a transaction
 
> or a reset raw usage change.

It seems that it's not recordererd as an actual transaction.. because actually it's a a transaction in the DB, the usage being reset is the one in scontrol.
But I would agree that this is confusing and that adding a transaction for it may make sense too.

> Is there any way to determine whether or not
> such an action occurred by some other
> method?

As you alerady did, you can check accounts "replacements" with "sacctmgr show transactions".
About reseting/decaying slurmctld usage with other commands can only be seen mainly in slurmctld logs, depending on the flags and log level.

> I think what I meant here was that: if we can't find the reason why it's 0
> could
> we set it again. I now know that we can't, but as you mentioned we still
> have a 
> mystery to solve because it shouldn't be 0 in our case (unless it was somehow
> deleted or had the usage set to 0; which I don't know if we can really query
> that at this point).

At this point, I also think that we are running out of ways to certainly solve the mistery.
And we cannot increase those usages neither.

> > An alternative would be actually to follow the Slurm way of thinking on
> > usage and limits.
> > That is, instead of (manually) changing/increasing limits, let Slurm to
> > (automatically) change/reduce usage (C, not D).
> > I think that you discarded this in some other issue, but now that you know
> > better about C and D, maybe that makes better sense now.
> > Maybe you don't want that automatic part, but you could still reset usages
> > to 0 instead of icreasing limits.
> > 
> > I totally understand your thinking on increasing limits, seems natural, but
> > note that in the long-long-long term increasing limits is problably not
> > technically possible, right?
> > That's also why Slurm uses the Decay usage idea.
> > So, you set a limit and you know that for a Decay/Reset period of time it
> > won't be violated, but after that period the old usage is not taken into
> > account. So, technically is like the limit has been increased.
> 
> This might be something that we could discuss internally; or at least a way
> to do this without increasing the limit ad-infinitum

I understand.
Please open a new support bug if you want to discuss it further with us too.

> I think given the scenarios you outlined (A-D) we can consider this stanza
> solved, 
> because I now understand that the values are independent for some time and
> sync up
> only at starts|restarts of the daemons, and that sshare/slurmctld are the
> better source of record for usages.

Yes.
Well, let me clarify this points, though:
1) Usage definitions are not totally "independet", ie. they are related but
   have differences that may lead to very different values.
2) Limits are always as sync as possible (if daemons are connected), not only
   on (re)starts.
3) Usages are also as sync as possible, but with their differences.
   And taking into account that sreport info is update only once per hour.

So, what happen in slurmctld and/or slurmdbd restarts is only that they both say to each other something like:
"Hey buddy! I missed you for a while... anyway, since the last time we contacted all this happen in my side (these jobs were submited, started, completed... this account was created, that qos changed...). Do you have any news for me in your side?"


Regards,
Albert

Hi David,

As you already closed bug 13400, I think that you will also agree on closing this one too as infogiven becasue the key info of both tickets is probably the comment 3 on this one.
Anyway, please don't hesitate to reopen it if you need further support.

Thanks,
Albert