Ticket 1744

I'm not able to reproduce this yet. Will you attach your slurm.conf and gres.conf?

Created attachment 1971 [details]
gres.conf

Created attachment 1972 [details]
slurm.conf

What does your lua job submit plugin do? I noticed this from your scontrol show jobs output, the job has:

   NumNodes=1 NumCPUs=2 CPUs/Task=16 ReqB:S:C:T=0:0:*:*
   Socks/Node=* NtasksPerN:B:S:C=0:0:*:1 CoreSpec=*
   MinCPUsNode=16 MinMemoryCPU=2000M MinTmpDiskNode=0

I'm looking at the 16 CPUs/Task and MinCPUsNode.

I can get my jobs into the same JobHeldAdmin scenario if I submit as:

sbatch --partition=gpu --gres=gpu:k20x:1 --ntasks=2 --ntasks-per-core=1 --wrap="sleep 60" -c16

Yes, that's exactly what our job submit plugin does. We want 8 cores (16 CPUs) to be allocated along with each GPU. Here's the relevant section from the plugin:

		elseif job_desc.partition == "gpu" then
			NIH_qos = "gpu"
			job_desc.cpus_per_task = 16

Thanks for that information. So you're request is then asking for 1 core with 16 cpus which doesn't exist. 

Are you trying to keep the tasks on one socket?

Do you need any more assistance on this?

Thanks for your explanation. We were setting cpus_per_task = 16 as a way to limit per-user allocation of GPUs, i.e. with each GPU, we allocated 16 CPUs and set a limit on total CPU allocation per user in the GPU queue. Now that we understand what's happening, we can find an alternate way to avoid using --ntasks-per-core=1,
and then we shouldn't hit this problem. 

When trackable resources are available we'll switch to using those.

Good to hear. One option is to have the jobs in the gpu queue allocate a full socket.

Here's the configuration to do that:
slurm.conf:
TaskPlugin=cgroup,affinity
SelectType=select/cons_res
SelectTypeParameters=CR_CORE_Memory,CR_ALLOCATE_FULL_SOCKET

PartitionName=gpu Nodes=cn[0603-0626] ... SelectTypeParameters=CR_SOCKET

and turn off TaskAffinity in the cgroup.conf since you are using the task/affinity plugin for task placement.


brian@compy:~/slurm/14.11/compy$ sbatch -p gpu -n3 -mblock:block --wrap="hostname"
Submitted batch job 432633
brian@compy:~/slurm/14.11/compy$ sbatch -p gpu -n13 -mblock:block --wrap="hostname"
Submitted batch job 432634

brian@compy:~/slurm/14.11/compy$ sacct -j 432633,432634
       JobID    JobName  Partition    Account  AllocCPUS      State ExitCode 
------------ ---------- ---------- ---------- ---------- ---------- -------- 
432633             wrap        gpu     normal         12  COMPLETED      0:0 
432633.batch      batch                normal         12  COMPLETED      0:0 
432634             wrap        gpu     normal         24  COMPLETED      0:0 
432634.batch      batch                normal         24  COMPLETED      0:0 


knc is 2 sockets with 12 cpus each. By default, tasks are allocated cyclically across sockets, so the block:block packs the tasks on one socket before allocating on the next socket. 



Let us know if you need any more help.

Thanks,
Brian