Ticket 16769

Hi,

If I understand correctly, you wish to submit jobs to GPU partition with only cpu requirements?  If a job requests memory, you wish to reject this job from submission?

Thanks for clarifying

Thanks Benny for your reply.

To your questions,

----If I understand correctly, you wish to submit jobs to GPU partition with only cpu requirements?  

Yes, we are trying to submit CPU-only jobs to the GPU partition

----If a job requests memory, you wish to reject this job from submission?

No, jobs to submit will require memory.

Thanks much for your help on this!

Regards,
Amy

Ok, thanks for the clarification.  Can you supply a job submit request that you wish to deny?  I just want to be sure as to the best way to guide you towards a solution.

Thanks

Hi Benny,

Here is a test job submission script that we used but it cannot be scheduled based on the current Slurm configurations.

#!/bin/bash
#SBATCH --job-name=testjob-serial
#SBATCH --output=%x.o%j
#SBATCH --partition=gpu
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --gres-flags=disable-binding
#SBATCH --mem-per-cpu=500MB  
module load gnu9
./testrun

When submitting this test job, it returned the following errors.

sbatch: error: The 'gpu' partition requires a minimum of one GPU/Gres per node: [-G,--gpus,--gress,--gpus-per-node]
sbatch: error: Batch job submission failed: Job size specification needs to be provided

Please let me know if you need other info. Thanks again for your help!


Amy

Ok thanks, I am just confused as to what you mean by "CPU-only jobs on the GPU node.".  If you can clarify this, it would be quite helpful.  Also, so I can get a full picture of your current configuration, can you please attach your current slurm.conf and gres.conf.

Thanks

Created attachment 30375 [details]
slurm.conf

Created attachment 30376 [details]
gres.conf

Please find the attached slurm.conf and gres.conf files.

The cpu-only job means the job only require CPU cores to run, not GPU.

Thanks,
Amy

Ok, I see.  Thanks again for sharing that information.  Do you have a job_submit filter in place by any chance?  If so, can you please share that as well?

Thanks

One option you have is to make one of your existing partitions include the GPU node, or create a separate partition that has the GPU node that is specifically for cpu-only jobs to use the GPU node when it's idle and then you can configure preemption for it to insure that GPU jobs have a priority.  

If you're not familiar with Slurm preemption, here is some documentation to get you started:

https://slurm.schedmd.com/preempt.html

If you decide to go down this path, I can explain more thoroughly how you could set this up but, for now, I just wanted to give you an introduction.

Another option is to set MaxTime in your "cpu-only" partition to allow so that it only allows short running jobs.  This way the main GPU partition never wait too long.

Please let me know if you have any questions regarding this.

Thanks

Hi Benny,

Thanks very much for your advice! Sounds good, I'll try to create a separate partition to allow CPU-only jobs. Then setup for the preemption. Will let you know if I end up with issues. 

Thanks,
Amy

You're welcome.  Let me know if you have any issues.

Thanks

Hi,

Just for completeness of this ticket, I will go ahead and close.  If you have any further questions or concerns please don't hesitate to re-open and we will be happy to assist you further.

Thanks