Ticket 17756

Summary:	use to job_submit.lua to set/change the default partition for scrontab job
Product:	Slurm	Reporter:	Charles Lu <Charles.Lu>
Component:	Configuration	Assignee:	Benjamin Witham <benjamin.witham>
Status:	RESOLVED INFOGIVEN	QA Contact:
Severity:	4 - Minor Issue
Priority:	---	CC:	benjamin.witham
Version:	23.02.2
Hardware:	Linux
OS:	Linux
Site:	NOAA	Slinky Site:	---
Alineos Sites:	---	Atos/Eviden Sites:	---
Confidential Site:	---	Coreweave sites:	---
Cray Sites:	---	DS9 clusters:	---
Google sites:	---	HPCnow Sites:	---
HPE Sites:	---	IBM Sites:	---
NOAA SIte:	GFDL	NoveTech Sites:	---
Nvidia HWinf-CS Sites:	---	OCF Sites:	---
Recursion Pharma Sites:	---	SFW Sites:	---
SNIC sites:	---	Tzag Elita Sites:	---
Linux Distro:	---	Machine Name:
CLE Version:		Version Fixed:
Target Release:	---	DevPrio:	---
Emory-Cloud Sites:	---

Description Charles Lu 2023-09-22 15:57:35 MDT

Here is the question.

As "job_submit/lua - expose a "cron_job" flag to identify jobs submitted
    through scrontab" in implemented in the new slurm version.

How can we use (is it possible) to use this with job_submit.lua to force the scrontab "task"/"job" to run in specific partition dedicated to run the scron?

In other words, ii the user use the wrong queue/partition in:

#SCRON -p wrong_partition
.....


Can we use job_submit.lua to modify/change the partition to the different one/dedicated one partition (scron_partition1) for scron jobs?

Assume we only have a cron_partition in this case.

If yes. can you provide us the sample code?

What about if we have more than one dedicated scron partitions?

Comment 1 Benjamin Witham 2023-09-25 14:19:00 MDT

Hello Charles, 

You should be able to direct your cron jobs to a selected cron_partition with something similar to this:

> function slurm_job_submit(job_desc, part_list, submit_uid)
>     if job_desc.cron_job then
>         job_desc.partition = "cron_partition"
>     end
>     return slurm.SUCCESS
> end
> 
> function slurm_job_modify(job_desc, job_rec, part_list, modify_uid)
>     return slurm.SUCCESS
> end

If you have currently running crontab jobs that you want to adhere to this new job_submit, you'll need to force the scrontab to resubmit them. The easiest way to do this is to run:

> scrontab

then add a blank line to the bottom. After saving and exiting, the jobs will be resubmitted and use this job_submit.

> What about if we have more than one dedicated scron partitions?

If you have more than one partition I would suggest having some way to differentiate the two. For example, this job_submit.lua has a high_core_cron_partition and a low_core_cron_partition. If a cron job requests more than 8 coresit will be placed in the high_core_cron_partition otherwise it will be placed in the low_core_cron_partition.

> function slurm_job_submit(job_desc, part_list, submit_uid)
>     if job_desc.cron_job then
>         if job_desc.cores > 8 then
>             job_desc.partition = "high_core_cron_partition"
>         else
>             job_desc.partition = "low_core_cron_partition"
>     end
>     return slurm.SUCCESS
> end
> 
> function slurm_job_modify(job_desc, job_rec, part_list, modify_uid)
>     return slurm.SUCCESS
> end

Comment 2 Charles Lu 2023-09-27 15:25:51 MDT

Ben, thx.

I am OOO until Oct 15th, is it okay to low the serverity of this case and keep it open until I return to office?

Comment 3 Benjamin Witham 2023-09-27 15:29:51 MDT

Of course. I'll decrease the priority of this ticket. Feel free to come back to this when you're able to.

Comment 4 Benjamin Witham 2023-10-17 10:48:07 MDT

Hello Charles, are you still having trouble with your scrontab jobs?

Comment 5 Charles Lu 2023-10-17 11:16:36 MDT

You may close the case. Thanks!

Regards,

Charles


On Tue, Oct 17, 2023 at 12:48 PM <bugs@schedmd.com> wrote:

> *Comment # 4 <https://bugs.schedmd.com/show_bug.cgi?id=17756#c4> on bug
> 17756 <https://bugs.schedmd.com/show_bug.cgi?id=17756> from Benjamin Witham
> <benjamin.witham@schedmd.com> *
>
> Hello Charles, are you still having trouble with your scrontab jobs?
>
> ------------------------------
> You are receiving this mail because:
>
>    - You reported the bug.
>
>

Comment 6 Benjamin Witham 2023-10-17 11:17:47 MDT

Closing.