Ticket 15672

Summary: Updating array jobs with sacctmgr when SLURM_ARRAY_JOB_ID=SLURM_JOB_ID
Product: Slurm Reporter: Josko Plazonic <plazonic>
Component: OtherAssignee: Jason Booth <jbooth>
Status: RESOLVED INFOGIVEN QA Contact:
Severity: 4 - Minor Issue    
Priority: ---    
Version: 22.05.6   
Hardware: Linux   
OS: Linux   
Site: Princeton (PICSciE) Alineos Sites: ---
Atos/Eviden Sites: --- Confidential Site: ---
Coreweave sites: --- Cray Sites: ---
DS9 clusters: --- HPCnow Sites: ---
HPE Sites: --- IBM Sites: ---
NOAA SIte: --- OCF Sites: ---
Recursion Pharma Sites: --- SFW Sites: ---
SNIC sites: --- Linux Distro: ---
Machine Name: CLE Version:
Version Fixed: Target Release: ---
DevPrio: --- Emory-Cloud Sites: ---

Description Josko Plazonic 2022-12-21 08:36:37 MST 
Hello,

currently when one does something like
 sacctmgr -i update job where jobid=$SLURM_JOB_ID set AdminComment=something
and $SLURM_ARRAY_JOB_ID==$SLURM_JOB_ID all of $SLURM_ARRAY_JOB_ID jobs will get that AdminComment set. There is no way to set *only* that jobid's AdminComment (or other similar fields) and leave other members of this array job untouched.

We would need either a way for sacctmgr to change only this one $SLURM_JOB_ID or maybe you should consider not reusing $SLURM_ARRAY_JOB_ID as a $SLURM_JOB_ID (this last solution was suggested by Tim when we met with him at SC22).

BTW, not sure which component this belongs to - I guess it depends how you want to handle it - so I entered it as "Other".

Josko
Comment 1 Jason Booth 2022-12-21 11:33:19 MST 
You need to use the JobID and not SLURM_ARRAY_JOB_ID for example:

>  sbatch -t 1 -a [1-4] --wrap="sleep 60"

I now have 4 jobs.

> JobId=5838 ArrayJobId=5837
> JobId=5839 ArrayJobId=5837
> JobId=5840 ArrayJobId=5837
> JobId=5837 ArrayJobId=5837


> sacct -j 5837 --format=jobid,AdminComment
> JobID           AdminComment
> ------------ ---------------
> 5837_1
> 5837_1.batch
> 5837_2                  test
> 5837_2.batch
> 5837_3
> 5837_3.batch
> 5837_4
> 5837_4.batch