Ticket 17756 - use to job_submit.lua to set/change the default partition for scrontab job
Summary: use to job_submit.lua to set/change the default partition for scrontab job
Status: RESOLVED INFOGIVEN
Alias: None
Product: Slurm
Classification: Unclassified
Component: Configuration (show other tickets)
Version: 23.02.2
Hardware: Linux Linux
: 4 - Minor Issue
Assignee: Benjamin Witham
QA Contact:
URL:
Depends on:
Blocks:
 
Reported: 2023-09-22 15:57 MDT by Charles Lu
Modified: 2023-10-17 11:17 MDT (History)
1 user (show)

See Also:
Site: NOAA
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: GFDL
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed:
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this ticket.
Description Charles Lu 2023-09-22 15:57:35 MDT 
Here is the question.

As "job_submit/lua - expose a "cron_job" flag to identify jobs submitted
    through scrontab" in implemented in the new slurm version.

How can we use (is it possible) to use this with job_submit.lua to force the scrontab "task"/"job" to run in specific partition dedicated to run the scron?

In other words, ii the user use the wrong queue/partition in:

#SCRON -p wrong_partition
.....


Can we use job_submit.lua to modify/change the partition to the different one/dedicated one partition (scron_partition1) for scron jobs?

Assume we only have a cron_partition in this case.

If yes. can you provide us the sample code?

What about if we have more than one dedicated scron partitions?
Comment 1 Benjamin Witham 2023-09-25 14:19:00 MDT 
Hello Charles, 

You should be able to direct your cron jobs to a selected cron_partition with something similar to this:

> function slurm_job_submit(job_desc, part_list, submit_uid)
>     if job_desc.cron_job then
>         job_desc.partition = "cron_partition"
>     end
>     return slurm.SUCCESS
> end
> 
> function slurm_job_modify(job_desc, job_rec, part_list, modify_uid)
>     return slurm.SUCCESS
> end

If you have currently running crontab jobs that you want to adhere to this new job_submit, you'll need to force the scrontab to resubmit them. The easiest way to do this is to run:

> scrontab

then add a blank line to the bottom. After saving and exiting, the jobs will be resubmitted and use this job_submit.

> What about if we have more than one dedicated scron partitions?

If you have more than one partition I would suggest having some way to differentiate the two. For example, this job_submit.lua has a high_core_cron_partition and a low_core_cron_partition. If a cron job requests more than 8 coresit will be placed in the high_core_cron_partition otherwise it will be placed in the low_core_cron_partition.

> function slurm_job_submit(job_desc, part_list, submit_uid)
>     if job_desc.cron_job then
>         if job_desc.cores > 8 then
>             job_desc.partition = "high_core_cron_partition"
>         else
>             job_desc.partition = "low_core_cron_partition"
>     end
>     return slurm.SUCCESS
> end
> 
> function slurm_job_modify(job_desc, job_rec, part_list, modify_uid)
>     return slurm.SUCCESS
> end
Comment 2 Charles Lu 2023-09-27 15:25:51 MDT 
Ben, thx.

I am OOO until Oct 15th, is it okay to low the serverity of this case and keep it open until I return to office?
Comment 3 Benjamin Witham 2023-09-27 15:29:51 MDT 
Of course. I'll decrease the priority of this ticket. Feel free to come back to this when you're able to.
Comment 4 Benjamin Witham 2023-10-17 10:48:07 MDT 
Hello Charles, are you still having trouble with your scrontab jobs?
Comment 5 Charles Lu 2023-10-17 11:16:36 MDT 
You may close the case. Thanks!

Regards,

Charles


On Tue, Oct 17, 2023 at 12:48 PM <bugs@schedmd.com> wrote:

> *Comment # 4 <https://bugs.schedmd.com/show_bug.cgi?id=17756#c4> on bug
> 17756 <https://bugs.schedmd.com/show_bug.cgi?id=17756> from Benjamin Witham
> <benjamin.witham@schedmd.com> *
>
> Hello Charles, are you still having trouble with your scrontab jobs?
>
> ------------------------------
> You are receiving this mail because:
>
>    - You reported the bug.
>
>
Comment 6 Benjamin Witham 2023-10-17 11:17:47 MDT 
Closing.