Ticket 15240

Paul, you can access this information today with a few of the sacct options.


https://slurm.schedmd.com/sacct.html#OPT_duplicates
https://slurm.schedmd.com/sacct.html#OPT_state


The -D option can be used with a specific job, so a user who wants to know how many times a job is restarted can use this option to count those events.

For a broader overview, you can use a timeframe with the state flag.

For example:
> sacct -S2020-01-01-9:00 -E2022-10-20-12:00 --state=PREEMPTED

https://slurm.schedmd.com/sacct.html#OPT_PR--PREEMPTED

> PR PREEMPTED
> Job terminated due to preemption.


Would these satisfy your needs?

That's quite a bit of work to just get a value for a counter.  A summary 
counter of how may requeues would be easier to reference and plot graphs 
of.  Imagine looping over all the jobs in your database to pull the 
requeue information for historic estimates of how many requeues on 
average jobs get.  Instead of querying a single integer value, you have 
to pull a bunch of information from the slurmdbd and then do some math 
to reconstruct the value.  Or if you want to poll this on a loop now you 
have massive sacct lines and returns coming for the slurmdbd to then 
reconstruct into a single value you want to plot.  This adds overhead to 
polling the scheduler rather than looking at a single counter per job 
that can be updated.

The data is already available easily in squeue as RestartCnt:

*RestartCnt*<https://slurm.schedmd.com/squeue.html#OPT_RestartCnt>
    The number of restarts for the job. (Valid for jobs only) 

Ideally all the values availble to squeue should be available as 
historic records in the slurmdbd.  This would also unify the information 
and options between squeue and sacct which arguably should be able to 
show the same data.

-PaulE dmon-

On 10/20/22 12:11 PM, bugs@schedmd.com wrote:
> Jason Booth <mailto:jbooth@schedmd.com> changed bug 15240 
> <https://bugs.schedmd.com/show_bug.cgi?id=15240>
> What 	Removed 	Added
> Assignee 	support@schedmd.com 	jbooth@schedmd.com
>
> *Comment # 2 <https://bugs.schedmd.com/show_bug.cgi?id=15240#c2> on 
> bug 15240 <https://bugs.schedmd.com/show_bug.cgi?id=15240> from Jason 
> Booth <mailto:jbooth@schedmd.com> *
> Paul, you can access this information today with a few of the sacct options.
>
>
> https://slurm.schedmd.com/sacct.html#OPT_duplicates
> https://slurm.schedmd.com/sacct.html#OPT_state
>
>
> The -D option can be used with a specific job, so a user who wants to know how
> many times a job is restarted can use this option to count those events.
>
> For a broader overview, you can use a timeframe with the state flag.
>
> For example:
> > sacct -S2020-01-01-9:00 -E2022-10-20-12:00 --state=PREEMPTED
>
> https://slurm.schedmd.com/sacct.html#OPT_PR--PREEMPTED
>
> > PR PREEMPTED > Job terminated due to preemption.
>
>
> Would these satisfy your needs?
> ------------------------------------------------------------------------
> You are receiving this mail because:
>
>   * You reported the bug.
>

I assume my reply was satisfactory. I am resolving this out. Please feel free to re-open this should you have question related to my last reply.