Ticket 9724

Summary: Memory requirement differs when comming from --mem-per-cpu for whole node if specific --exclusive added
Product: Slurm Reporter: Marcin Stolarek <cinek>
Component: SchedulingAssignee: Marcin Stolarek <cinek>
Status: RESOLVED FIXED QA Contact:
Severity: 4 - Minor Issue    
Priority: --- CC: csc-slurm-tickets, gensyshpe, marc.caubet, marshall, remi.lacroix, slakkaraju
Version: 20.02.4   
Hardware: Linux   
OS: Linux   
See Also: https://bugs.schedmd.com/show_bug.cgi?id=13879
Site: GENCI - Grand Equipement National de Calcul Intensif Slinky Site: ---
Alineos Sites: --- Atos/Eviden Sites: ---
Confidential Site: --- Coreweave sites: ---
Cray Sites: --- DS9 clusters: ---
Google sites: --- HPCnow Sites: ---
HPE Sites: --- IBM Sites: ---
NOAA SIte: --- NoveTech Sites: ---
Nvidia HWinf-CS Sites: --- OCF Sites: ---
Recursion Pharma Sites: --- SFW Sites: ---
SNIC sites: --- Tzag Elita Sites: ---
Linux Distro: --- Machine Name:
CLE Version: Version Fixed: 20.11.0
Target Release: --- DevPrio: ---
Emory-Cloud Sites: ---

Description Marcin Stolarek 2020-09-02 11:04:06 MDT 
Splitting this from Bug 9670

In the case of CR_Core_Memory and --hint=nomultithread (--threads-per-core=1) the job memory required per node is calculated differently for whole node allocation vs specific --exclusive added to srun.

This doesn't happen for CR_CPU (CPUs are consumable resources so we "charge" only for used threads, because --threads-per-core=1 makes use of only one thread per core total number of CPUs is half the number of those in case of CR_Core.

Node configuration:
NodeName=DEFAULT RealMemory=900 CPUS=72 Sockets=2 CoresPerSocket=18 ThreadsPerCore=2

1)SelectTypeParameters    = CR_CPU_MEMORY
# srun --mem-per-cpu=10 -n2  -c18 --exclusive --hint=nomultithread /bin/bash -c "scontrol show job $SLURM_JOB_ID | grep  mem; echo STEP; scontrol show step $SLURM_JOB_ID | grep mem"
   TRES=cpu=72,mem=360M,node=1,billing=72
   TRES=cpu=72,mem=360M,node=1,billing=72
STEP
STEP
   TRES=cpu=36,mem=360M,node=1
   TRES=cpu=36,mem=360M,node=1

# srun --mem-per-cpu=10 -n2  -c18  --hint=nomultithread /bin/bash -c "scontrol show job $SLURM_JOB_ID | grep  mem; echo STEP; scontrol show step $SLURM_JOB_ID | grep mem"
   TRES=cpu=36,mem=360M,node=1,billing=36
   TRES=cpu=36,mem=360M,node=1,billing=36
STEP
STEP
   TRES=cpu=36,mem=360M,node=1
   TRES=cpu=36,mem=360M,node=1

2)SelectTypeParameters = CR_Core_Memory
***Job memory for non-exclusive allocation is higher than step***
# srun --mem-per-cpu=10 -n2  -c18  --hint=nomultithread /bin/bash -c "scontrol show job $SLURM_JOB_ID | grep  mem; echo STEP; scontrol show step $SLURM_JOB_ID | grep mem"
   TRES=cpu=72,mem=720M,node=1,billing=72
   TRES=cpu=72,mem=720M,node=1,billing=72
STEP
STEP
   TRES=cpu=36,mem=360M,node=1
   TRES=cpu=36,mem=360M,node=1

# srun --mem-per-cpu=10 -n2 --exclusive  -c18  --hint=nomultithread /bin/bash -c "scontrol show job $SLURM_JOB_ID | grep  mem; echo STEP; scontrol show step $SLURM_JOB_ID | grep mem"
   TRES=cpu=72,mem=360M,node=1,billing=72
STEP
   TRES=cpu=72,mem=360M,node=1,billing=72
STEP
   TRES=cpu=36,mem=360M,node=1
   TRES=cpu=36,mem=360M,node=1
Comment 7 Marcin Stolarek 2020-11-05 06:09:47 MST 
I have a patch altering this behavior - adjusting number of allocated CPUs to specified --threads-per-core (used behind the scene by --hint=nomultithread to request 1 thread per core).

I'm passing it to the review queue now. Let me know if you want to give it a try locally.

cheers,
Marcin
Comment 21 Marshall Garey 2020-11-10 14:59:03 MST 
*** Ticket 9931 has been marked as a duplicate of this ticket. ***
Comment 28 Marshall Garey 2020-11-13 12:41:51 MST 
*** Ticket 9153 has been marked as a duplicate of this ticket. ***
Comment 30 Marcin Stolarek 2020-11-16 04:14:56 MST 
Handling of the case in comment 0 should be improved by 49a7d7f9fb, which will be part of 20.11.0 release.

cheers,
Marcin
Comment 31 Marshall Garey 2020-11-20 14:51:37 MST 
*** Ticket 10262 has been marked as a duplicate of this ticket. ***
Comment 32 CSC sysadmins 2020-11-23 04:57:49 MST 
Hi,

Could you commit this fix 49a7d7f9fb to 20.02 branch? I applied it to our 20.02.6 build and it solved the scheduling problem.
Comment 33 Marcin Stolarek 2020-11-23 05:19:08 MST 
Tommi,

The change in behavior made in the related commit is probably reasonable for the majority of users, but since it's not a clear bug fix but a change we don't want to backport it to older major releases.
If it's feasible for you you can always keep the commit in your tree, it's a one place change that shouldn't conflict with other changes happening on 20.02.

cheers,
Marcin
Comment 34 Marshall Garey 2021-01-08 08:40:54 MST 
*** Ticket 10434 has been marked as a duplicate of this ticket. ***