Ticket 2682

Summary:	job_submit.lua and job arrays
Product:	Slurm	Reporter:	rl303f
Component:	Other	Assignee:	Tim Wickberg <tim>
Status:	RESOLVED FIXED	QA Contact:
Severity:	4 - Minor Issue
Priority:	---
Version:	15.08.10
Hardware:	Linux
OS:	Linux
Site:	NIH	Alineos Sites:	---
Atos/Eviden Sites:	---	Confidential Site:	---
Coreweave sites:	---	Cray Sites:	---
DS9 clusters:	---	HPCnow Sites:	---
HPE Sites:	---	IBM Sites:	---
NOAA SIte:	---	OCF Sites:	---
Recursion Pharma Sites:	---	SFW Sites:	---
SNIC sites:	---	Linux Distro:	---
Machine Name:		CLE Version:
Version Fixed:	15.08.12 16.05.0-rc2	Target Release:	---
DevPrio:	---	Emory-Cloud Sites:	---
Attachments:	add array_inx to job_submit.lua

Description rl303f 2016-05-02 03:48:51 MDT

We are trying to find a way for job_submit.lua to detect
that a submit request is for a job array.

I've been going through the source code trying to find
which 'job_des.XXXXX' variable can be tested by job_submit.lua
to make this determination but have not been successful.

Can you provide a quick pointer as to how this may be accomplished?

Thank you!

Comment 1 Tim Wickberg 2016-05-02 04:26:25 MDT

(In reply to rl303f from comment #0)
> We are trying to find a way for job_submit.lua to detect
> that a submit request is for a job array.

Unfortunately, it doesn't look like anyone has previously asked for those fields to be made available, so they're not exposed on the Lua interface currently.

I'm assuming that having access to the raw array string would be sufficient - i.e., the "1-100%2" part of an "sbatch --array=1-100%2 job_script.sh"?

I can look at getting a patch for you shortly.

> I've been going through the source code trying to find
> which 'job_des.XXXXX' variable can be tested by job_submit.lua
> to make this determination but have not been successful.

There's no separate documentation on this at present, the best reference is the plugin itself. The job description fields are all constructed here:

https://github.com/SchedMD/slurm/blob/master/src/plugins/job_submit/lua/job_submit_lua.c#L651

FYI - There's separate code to validate and update the record later, there's usually three locations any patch has to address if you're looking at modifying it yourself. I think the field you care to see is array_task_str.

- Tim

Comment 2 rl303f 2016-05-03 00:06:10 MDT

Yes, the raw array string would be great.  Thanks very much, Tim!

Comment 3 Tim Wickberg 2016-05-10 05:47:31 MDT

Created attachment 3072 [details]
add array_inx to job_submit.lua

Comment 4 Tim Wickberg 2016-05-10 06:05:44 MDT

This is added in commit f52a331769. I attached a standalone patch to the bug as well.

This will be in 15.08.12 / 16.05-rc2.

The value is 'job_desc.array_inx', and needs a properly formatted job array string if you're going to set it for people. If you're just testing it, it is 'nil' (null) if unset, or a string if set on the submission.

Note that this only works for submission, not for modification. (Slurm as a whole does not provide a method for modifying the overall job array at present.)

- Tim

Comment 5 rl303f 2016-05-10 09:42:15 MDT

Wow, that's perfect!  Many thanks, Tim.