I noticed that we're getting the wrong number of CPUs under certain conditions. Let's say we have two partitions, gpu1 and gpu2. They have different DefCpuPerGPU values: PartitionName=gpu1 Nodes=gpu[12-21] OverSubscribe=NO JobDefaults=DefCpuPerGPU=8 [...] PartitionName=gpu2 Nodes=gpu[22-33] OverSubscribe=NO JobDefaults=DefCpuPerGPU=12 [...] Now I run a job that could start in either of the two partitions, requesting one GPU. The first partition doesn't have free nodes, so the jobs runs on a node from gpu2. So far so good, but the number of CPUs allocated is wrong. $ srun -p gpu1,gpu2 -G 1 bash -c 'hostname; nproc;' srun: Required node not available (down, drained or reserved) srun: job 5271 queued and waiting for resources srun: job 5271 has been allocated resources gpu25 8 Simply changing the order of partitions gives me the correct number: $ srun -p gpu2,gpu1 -G 1 bash -c 'hostname; nproc;' srun: job 5272 queued and waiting for resources srun: job 5272 has been allocated resources gpu25 12 So apparently the DefCpuPerGPU setting from the partition is used before the scheduler has decided in which partition the job will actually run? Or is this a problem with sched/backfill? I've tested with Slurm 22.05.3 and 22.05.5. Important settings from slurm.conf: SchedulerType=sched/backfill SelectType=select/cons_tres SelectTypeParameters=CR_Core_Memory,CR_ONE_TASK_PER_CORE,CR_CORE_DEFAULT_DIST_BLOCK
while the job is still in pending state it looks like the correct value is picked up (CpusPerTres=gpu:12) JobId=5271 JobName=bash JobState=PENDING Reason=Resources Dependency=(null) Requeue=1 Restarts=0 BatchFlag=0 Reboot=0 ExitCode=0:0 RunTime=00:00:00 TimeLimit=08:00:00 TimeMin=N/A SubmitTime=2022-10-19T15:17:56 EligibleTime=2022-10-19T15:17:56 AccrueTime=2022-10-19T15:17:56 StartTime=Unknown EndTime=Unknown Deadline=N/A Partition=gpu1,gpu2 NodeList= NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* TRES=cpu=1,mem=1G,node=1,billing=1 Socks/Node=* NtasksPerN:B:S:C=0:0:*:1 CoreSpec=* MinCPUsNode=1 MinMemoryCPU=1G MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null) Command=bash CpusPerTres=gpu:12 MemPerTres=gpu:16384 TresPerJob=gres:gpu:1 [...] After execution, still CpusPerTres=gpu:12 but TRES=cpu=8 JobId=5271 JobName=bash JobState=COMPLETED Reason=None Dependency=(null) Requeue=1 Restarts=0 BatchFlag=0 Reboot=0 ExitCode=0:0 RunTime=00:00:00 TimeLimit=08:00:00 TimeMin=N/A Partition=gpu2 ReqNodeList=(null) ExcNodeList=(null) NodeList=gpu25 BatchHost=gpu25 NumNodes=1 NumCPUs=8 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* TRES=cpu=8,mem=16G,node=1,billing=8 Socks/Node=* NtasksPerN:B:S:C=0:0:*:1 CoreSpec=* MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null) Command=bash CpusPerTres=gpu:12 MemPerTres=gpu:16384 TresPerJob=gres:gpu:1