Ticket 9724 - Memory requirement differs when comming from --mem-per-cpu for whole node if specific --exclusive added
Summary: Memory requirement differs when comming from --mem-per-cpu for whole node if ...
Status: RESOLVED FIXED
Alias: None
Product: Slurm
Classification: Unclassified
Component: Scheduling (show other tickets)
Version: 20.02.4
Hardware: Linux Linux
: 4 - Minor Issue
Assignee: Marcin Stolarek
QA Contact:
URL:
: 9153 9931 10262 10434 (view as ticket list)
Depends on:
Blocks:
 
Reported: 2020-09-02 11:04 MDT by Marcin Stolarek
Modified: 2022-04-22 18:18 MDT (History)
6 users (show)

See Also:
Site: GENCI - Grand Equipement National de Calcul Intensif
Slinky Site: ---
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
Google sites: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
NoveTech Sites: ---
Nvidia HWinf-CS Sites: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Tzag Elita Sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed: 20.11.0
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this ticket.
Description Marcin Stolarek 2020-09-02 11:04:06 MDT 
Splitting this from Bug 9670

In the case of CR_Core_Memory and --hint=nomultithread (--threads-per-core=1) the job memory required per node is calculated differently for whole node allocation vs specific --exclusive added to srun.

This doesn't happen for CR_CPU (CPUs are consumable resources so we "charge" only for used threads, because --threads-per-core=1 makes use of only one thread per core total number of CPUs is half the number of those in case of CR_Core.

Node configuration:
NodeName=DEFAULT RealMemory=900 CPUS=72 Sockets=2 CoresPerSocket=18 ThreadsPerCore=2

1)SelectTypeParameters    = CR_CPU_MEMORY
# srun --mem-per-cpu=10 -n2  -c18 --exclusive --hint=nomultithread /bin/bash -c "scontrol show job $SLURM_JOB_ID | grep  mem; echo STEP; scontrol show step $SLURM_JOB_ID | grep mem"
   TRES=cpu=72,mem=360M,node=1,billing=72
   TRES=cpu=72,mem=360M,node=1,billing=72
STEP
STEP
   TRES=cpu=36,mem=360M,node=1
   TRES=cpu=36,mem=360M,node=1

# srun --mem-per-cpu=10 -n2  -c18  --hint=nomultithread /bin/bash -c "scontrol show job $SLURM_JOB_ID | grep  mem; echo STEP; scontrol show step $SLURM_JOB_ID | grep mem"
   TRES=cpu=36,mem=360M,node=1,billing=36
   TRES=cpu=36,mem=360M,node=1,billing=36
STEP
STEP
   TRES=cpu=36,mem=360M,node=1
   TRES=cpu=36,mem=360M,node=1

2)SelectTypeParameters = CR_Core_Memory
***Job memory for non-exclusive allocation is higher than step***
# srun --mem-per-cpu=10 -n2  -c18  --hint=nomultithread /bin/bash -c "scontrol show job $SLURM_JOB_ID | grep  mem; echo STEP; scontrol show step $SLURM_JOB_ID | grep mem"
   TRES=cpu=72,mem=720M,node=1,billing=72
   TRES=cpu=72,mem=720M,node=1,billing=72
STEP
STEP
   TRES=cpu=36,mem=360M,node=1
   TRES=cpu=36,mem=360M,node=1

# srun --mem-per-cpu=10 -n2 --exclusive  -c18  --hint=nomultithread /bin/bash -c "scontrol show job $SLURM_JOB_ID | grep  mem; echo STEP; scontrol show step $SLURM_JOB_ID | grep mem"
   TRES=cpu=72,mem=360M,node=1,billing=72
STEP
   TRES=cpu=72,mem=360M,node=1,billing=72
STEP
   TRES=cpu=36,mem=360M,node=1
   TRES=cpu=36,mem=360M,node=1
Comment 7 Marcin Stolarek 2020-11-05 06:09:47 MST 
I have a patch altering this behavior - adjusting number of allocated CPUs to specified --threads-per-core (used behind the scene by --hint=nomultithread to request 1 thread per core).

I'm passing it to the review queue now. Let me know if you want to give it a try locally.

cheers,
Marcin
Comment 21 Marshall Garey 2020-11-10 14:59:03 MST 
*** Ticket 9931 has been marked as a duplicate of this ticket. ***
Comment 28 Marshall Garey 2020-11-13 12:41:51 MST 
*** Ticket 9153 has been marked as a duplicate of this ticket. ***
Comment 30 Marcin Stolarek 2020-11-16 04:14:56 MST 
Handling of the case in comment 0 should be improved by 49a7d7f9fb, which will be part of 20.11.0 release.

cheers,
Marcin
Comment 31 Marshall Garey 2020-11-20 14:51:37 MST 
*** Ticket 10262 has been marked as a duplicate of this ticket. ***
Comment 32 CSC sysadmins 2020-11-23 04:57:49 MST 
Hi,

Could you commit this fix 49a7d7f9fb to 20.02 branch? I applied it to our 20.02.6 build and it solved the scheduling problem.
Comment 33 Marcin Stolarek 2020-11-23 05:19:08 MST 
Tommi,

The change in behavior made in the related commit is probably reasonable for the majority of users, but since it's not a clear bug fix but a change we don't want to backport it to older major releases.
If it's feasible for you you can always keep the commit in your tree, it's a one place change that shouldn't conflict with other changes happening on 20.02.

cheers,
Marcin
Comment 34 Marshall Garey 2021-01-08 08:40:54 MST 
*** Ticket 10434 has been marked as a duplicate of this ticket. ***