Ticket 13584

Hello,

One of our users reported a resource allocation issue which I cannot explain, while mixing exclusive steps and overlapping steps.

Here is a minimal example to reproduce this issue:

#! /bin/bash -l
#SBATCH -N 1
#SBATCH --ntasks-per-node=7
#SBATCH --cpus-per-task 1
#SBATCH --time=0-00:30:00
#SBATCH --partition interactive
SRUN=(srun --exclusive --ntasks=1 --cpus-per-task=1)
PARALLEL=( \
        parallel \
        --delay .2 \
        --jobs "${SLURM_NTASKS}" \
        --joblog parallel.log \
        --line-buffer \
)
NTASKS=1000
"${PARALLEL[@]}" "${SRUN[@]}" "./exec.sh" {} :::: <(seq "${NTASKS}")



The script "exec.sh" called above displays the cpu affinity of all exec.sh processes started via "srun --exclusive" at a given time.

#! /bin/bash
echo "PID $$: Starting, CPU affinities:"
for pid in $(pgrep exec.sh) ; do
        taskset -cp $pid
done
DELAY=$(( (RANDOM % 5) + 2 ))
echo "PID $$: Sleeping for ${DELAY}s"
sleep "$DELAY"


In the beginning, everything is fine, each process runs on a different cpu core:

PID 232353: Starting, CPU affinities:
pid 232217's current affinity list: 102
pid 232238's current affinity list: 59
pid 232259's current affinity list: 63
pid 232281's current affinity list: 67
pid 232304's current affinity list: 71
pid 232337's current affinity list: 75
pid 232353's current affinity list: 79
PID 232353: Sleeping for 2s
PID 232444: Starting, CPU affinities:
pid 232238's current affinity list: 59
pid 232259's current affinity list: 63
pid 232281's current affinity list: 67
pid 232304's current affinity list: 71
pid 232337's current affinity list: 75
pid 232353's current affinity list: 79
pid 232444's current affinity list: 102
PID 232444: Sleeping for 2s


As soon as we start one overlapping step, in example with this command used for debugging purposes (srun --jobid <JOB_ID> --overlap --gres=gpu:0  --pty bash -i), the resource allocated previously to this overlapping step start being allocated for multiple exclusive steps at the same time (in the following extract, the cpu 102).


PID 234869: Starting, CPU affinities:
pid 234619's current affinity list: 79
pid 234661's current affinity list: 59
pid 234709's current affinity list: 67
pid 234784's current affinity list: 102
pid 234821's current affinity list: 75
pid 234844's current affinity list: 71
pid 234869's current affinity list: 102
PID 234869: Sleeping for 6s


Is this an expected behavior ?

Thank you for your support,

Best regards,

Hyacinthe