Hello,


we have identified a strange problem with running multi-threaded code on our cluster. In short, we allocate a number of threads for a task and run more threads than the cores allocated. It works fine at first, but after a few minutes, we can see that the threads are running on cores that were not allowed. Here is how to reproduce:

[user@maestro-submit ~]$ salloc -c 1
salloc: Pending job allocation 53430232
salloc: job 53430232 queued and waiting for resources
salloc: job 53430232 has been allocated resources
salloc: Granted job allocation 53430232
salloc: Waiting for resource configuration
salloc: Nodes maestro-1041 are ready for job

[user@maestro-1041 ~]$ srun bash -c "dd if=/dev/zero status=progress bs=1M count=102400 | pigz -p 16 -c > /dev/null"
959^Csrun: interrupt (one more within 1 sec to abort)MB/s 
srun: StepId=53430232.0 task 0: running
<running here>

so far so good, we can check that they all run on one core:
[user@maestro-1041 ~]$ ps -L -C pigz --no-headers -o psr
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8

but after few minutes:
[user@maestro-1041 ~]$ ps -L -C pigz --no-headers -o psr
  6
 22
 28
 30
 19
 41
 24
 26
 42
 23
 33
 12
 18
 14
 34
 29
 25
 40


this kind of behavior can be observed with other threaded programs (sysbench, mrand) but not with fork().

here is slurm cgroup file:
# CPUs
ConstrainCores=yes

# GPUS
ConstrainDevices=yes

# Memory
ConstrainRAMSpace=yes

and the relevant part of slurm.conf

ProctrackType=proctrack/cgroup
TaskPlugin=task/cgroup
TaskPluginParam=SlurmdOffSpec

thanks in advance for looking into it.

best regards,
   Konstantin