Ticket 16400

Summary:	Grab scontrol show job after job is finished
Product:	Slurm	Reporter:	Aravind <aravind.padmanabhan>
Component:	Accounting	Assignee:	Director of Support <support>
Status:	RESOLVED INFOGIVEN	QA Contact:
Severity:	4 - Minor Issue
Priority:	---	CC:	benny
Version:	22.05.2
Hardware:	Linux
OS:	Linux
Site:	Sick Kids	Alineos Sites:	---
Atos/Eviden Sites:	---	Confidential Site:	---
Coreweave sites:	---	Cray Sites:	---
DS9 clusters:	---	HPCnow Sites:	---
HPE Sites:	---	IBM Sites:	---
NOAA SIte:	---	OCF Sites:	---
Recursion Pharma Sites:	---	SFW Sites:	---
SNIC sites:	---	Linux Distro:	---
Machine Name:		CLE Version:
Version Fixed:		Target Release:	---
DevPrio:	---	Emory-Cloud Sites:	---

Description Aravind 2023-03-29 10:23:26 MDT

HI.
Not sure if this has been asked before. 
I would like to get a similar output as that of scontrol show job <id> when the job is finished.
We have implemented the  AccountingStoreFlags=job_script to save the job script to help us debug.

Is there a way to dump the information about the job once finished.
Thank you

Comment 1 Benny Hedayati 2023-03-29 10:56:12 MDT

Hi,

What information exactly are you wishing to grab from scontol show job once it is finished?  Have you considered using sacct command see here?

https://slurm.schedmd.com/sacct.html

There are many metrics that can be obtained from sacct from completed jobs.

Thanks

Comment 2 Aravind 2023-03-29 11:05:13 MDT

Hi Benny,
Thank you for reaching out.
In my hurry I forgot to add that information. Yes, I tried sacct but not sure of what options i should be using.
Basically, I was wondering if it would be possible to get historic information like when the job started, if it was pending (if so reason). etc.
Thanks

Comment 3 Benny Hedayati 2023-03-30 12:16:47 MDT

Hi,

Indeed you can get that information using sacct.

$ sacct -a -o <list_of_fields>

will give you the list of all the jobs that have ran and are currently running.

You have a number of fields to choose from including start and state you can review those here:

https://slurm.schedmd.com/sacct.html

Also you can go back in time and review jobs from the past by setting start and end times like so:

$ sacct -S <start_time> -E <end_time> -o <list_of_fields>

Note that jobs from the past that have ended will note include pending or running state information.

The "planned" field seen here:

https://slurm.schedmd.com/sacct.html#OPT_Planned

Will let you know how long a job stayed in the key.

Also using Slurm environment variables in you epilog script seen here:

https://slurm.schedmd.com/prolog_epilog.html

Can also give you useful information once job has terminated.

Thanks

Comment 4 Benny Hedayati 2023-04-03 07:07:52 MDT

Hi,

Do you have any further questions regarding this ticket?  If not can we go ahead and close it?

Thanks

Comment 5 Aravind 2023-04-03 07:45:35 MDT

Hi Benny,
It sorts of helps us. We have shifted our priorities. So I think best to close now and then I can come back once I'm working on this again.
Thank you so much for your help
Have an awesome rest of the week!

Comment 6 Benny Hedayati 2023-04-03 07:53:49 MDT

Ok sounds good.  Please let us know if you have any concerns.  Good week to you as well.

Thanks