Ticket 16400 - Grab scontrol show job after job is finished
Summary: Grab scontrol show job after job is finished
Status: RESOLVED INFOGIVEN
Alias: None
Product: Slurm
Classification: Unclassified
Component: Accounting (show other tickets)
Version: 22.05.2
Hardware: Linux Linux
: 4 - Minor Issue
Assignee: Director of Support
QA Contact:
URL:
Depends on:
Blocks:
 
Reported: 2023-03-29 10:23 MDT by Aravind
Modified: 2023-04-03 07:53 MDT (History)
1 user (show)

See Also:
Site: Sick Kids
Slinky Site: ---
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
Google sites: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
NoveTech Sites: ---
Nvidia HWinf-CS Sites: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Tzag Elita Sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed:
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---


Attachments

Note You need to log in before you can comment on or make changes to this ticket.
Description Aravind 2023-03-29 10:23:26 MDT
HI.
Not sure if this has been asked before. 
I would like to get a similar output as that of scontrol show job <id> when the job is finished.
We have implemented the  AccountingStoreFlags=job_script to save the job script to help us debug.

Is there a way to dump the information about the job once finished.
Thank you
Comment 1 Benny Hedayati 2023-03-29 10:56:12 MDT
Hi,

What information exactly are you wishing to grab from scontol show job once it is finished?  Have you considered using sacct command see here?

https://slurm.schedmd.com/sacct.html

There are many metrics that can be obtained from sacct from completed jobs.

Thanks
Comment 2 Aravind 2023-03-29 11:05:13 MDT
Hi Benny,
Thank you for reaching out.
In my hurry I forgot to add that information. Yes, I tried sacct but not sure of what options i should be using.
Basically, I was wondering if it would be possible to get historic information like when the job started, if it was pending (if so reason). etc.
Thanks
Comment 3 Benny Hedayati 2023-03-30 12:16:47 MDT
Hi,

Indeed you can get that information using sacct.

$ sacct -a -o <list_of_fields>

will give you the list of all the jobs that have ran and are currently running.

You have a number of fields to choose from including start and state you can review those here:

https://slurm.schedmd.com/sacct.html

Also you can go back in time and review jobs from the past by setting start and end times like so:

$ sacct -S <start_time> -E <end_time> -o <list_of_fields>

Note that jobs from the past that have ended will note include pending or running state information.

The "planned" field seen here:

https://slurm.schedmd.com/sacct.html#OPT_Planned

Will let you know how long a job stayed in the key.

Also using Slurm environment variables in you epilog script seen here:

https://slurm.schedmd.com/prolog_epilog.html

Can also give you useful information once job has terminated.

Thanks
Comment 4 Benny Hedayati 2023-04-03 07:07:52 MDT
Hi,

Do you have any further questions regarding this ticket?  If not can we go ahead and close it?

Thanks
Comment 5 Aravind 2023-04-03 07:45:35 MDT
Hi Benny,
It sorts of helps us. We have shifted our priorities. So I think best to close now and then I can come back once I'm working on this again.
Thank you so much for your help
Have an awesome rest of the week!
Comment 6 Benny Hedayati 2023-04-03 07:53:49 MDT
Ok sounds good.  Please let us know if you have any concerns.  Good week to you as well.

Thanks