Ticket 15238

Summary: wrong number of CPUs allocated when using partitions with different DefCpuPerGPU setting
Product: Slurm Reporter: Stefan <stefan>
Component: SchedulingAssignee: Jacob Jenson <jacob>
Status: RESOLVED INVALID QA Contact:
Severity: 6 - No support contract    
Priority: ---    
Version: 22.05.5   
Hardware: Linux   
OS: Linux   
Site: -Other- Slinky Site: ---
Alineos Sites: --- Atos/Eviden Sites: ---
Confidential Site: --- Coreweave sites: ---
Cray Sites: --- DS9 clusters: ---
Google sites: --- HPCnow Sites: ---
HPE Sites: --- IBM Sites: ---
NOAA SIte: --- NoveTech Sites: ---
Nvidia HWinf-CS Sites: --- OCF Sites: ---
Recursion Pharma Sites: --- SFW Sites: ---
SNIC sites: --- Tzag Elita Sites: ---
Linux Distro: --- Machine Name:
CLE Version: Version Fixed:
Target Release: --- DevPrio: ---
Emory-Cloud Sites: ---

Description Stefan 2022-10-19 07:16:17 MDT
I noticed that we're getting the wrong number of CPUs under certain conditions.
Let's say we have two partitions, gpu1 and gpu2. They have different DefCpuPerGPU values:

PartitionName=gpu1
   Nodes=gpu[12-21]
   OverSubscribe=NO
   JobDefaults=DefCpuPerGPU=8
   [...]

PartitionName=gpu2
   Nodes=gpu[22-33]
   OverSubscribe=NO
   JobDefaults=DefCpuPerGPU=12
   [...]


Now I run a job that could start in either of the two partitions, requesting one GPU. The first partition doesn't have free nodes, so the jobs runs on a node from gpu2. So far so good, but the number of CPUs allocated is wrong.

 $ srun -p gpu1,gpu2 -G 1 bash -c 'hostname; nproc;'
 srun: Required node not available (down, drained or reserved)
 srun: job 5271 queued and waiting for resources
 srun: job 5271 has been allocated resources
 gpu25
 8

Simply changing the order of partitions gives me the correct number:
 $ srun -p gpu2,gpu1 -G 1 bash -c 'hostname; nproc;'
 srun: job 5272 queued and waiting for resources
 srun: job 5272 has been allocated resources
 gpu25
 12

So apparently the DefCpuPerGPU setting from the partition is used before the scheduler has decided in which partition the job will actually run? Or is this a problem with sched/backfill?

I've tested with Slurm 22.05.3 and 22.05.5. Important settings from slurm.conf:
SchedulerType=sched/backfill
SelectType=select/cons_tres
SelectTypeParameters=CR_Core_Memory,CR_ONE_TASK_PER_CORE,CR_CORE_DEFAULT_DIST_BLOCK
Comment 1 Stefan 2022-10-19 07:24:11 MDT
while the job is still in pending state it looks like the correct value is picked up (CpusPerTres=gpu:12)

JobId=5271 JobName=bash
   JobState=PENDING Reason=Resources Dependency=(null)
   Requeue=1 Restarts=0 BatchFlag=0 Reboot=0 ExitCode=0:0
   RunTime=00:00:00 TimeLimit=08:00:00 TimeMin=N/A
   SubmitTime=2022-10-19T15:17:56 EligibleTime=2022-10-19T15:17:56
   AccrueTime=2022-10-19T15:17:56
   StartTime=Unknown EndTime=Unknown Deadline=N/A
   Partition=gpu1,gpu2
   NodeList=
   NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
   TRES=cpu=1,mem=1G,node=1,billing=1
   Socks/Node=* NtasksPerN:B:S:C=0:0:*:1 CoreSpec=*
   MinCPUsNode=1 MinMemoryCPU=1G MinTmpDiskNode=0
   Features=(null) DelayBoot=00:00:00
   OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
   Command=bash
   CpusPerTres=gpu:12
   MemPerTres=gpu:16384
   TresPerJob=gres:gpu:1
   [...]


After execution, still CpusPerTres=gpu:12 but TRES=cpu=8
JobId=5271 JobName=bash
   JobState=COMPLETED Reason=None Dependency=(null)
   Requeue=1 Restarts=0 BatchFlag=0 Reboot=0 ExitCode=0:0
   RunTime=00:00:00 TimeLimit=08:00:00 TimeMin=N/A
   Partition=gpu2
   ReqNodeList=(null) ExcNodeList=(null)
   NodeList=gpu25
   BatchHost=gpu25
   NumNodes=1 NumCPUs=8 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
   TRES=cpu=8,mem=16G,node=1,billing=8
   Socks/Node=* NtasksPerN:B:S:C=0:0:*:1 CoreSpec=*
   MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
   Features=(null) DelayBoot=00:00:00
   OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
   Command=bash
   CpusPerTres=gpu:12
   MemPerTres=gpu:16384
   TresPerJob=gres:gpu:1