Ticket 11275

possibly related to the same issue, if I use the following script SLRUM will start 28 tasks on first node, but only single task on second node allocated to job:

#!/bin/bash
#SBATCH --ntasks 40
#SBATCH --partition linlarge
#SBATCH --exclusive

for i in {1..40}; do
echo $i
srun -n 1 -c 1 --exclusive sleep 150 &
done

wait

             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
            677547  linlarge slurmtes  ngib740  R       2:56      2 gisath[018,020]

 cat slurm-677547.out
cpu-bind=MASK - gisath018, task  0  0 [14335]: mask 0xfffffff set
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
cpu-bind=MASK - gisath018, task  0  0 [14584]: mask 0x2 set
cpu-bind=MASK - gisath018, task  0  0 [14591]: mask 0x4 set
cpu-bind=MASK - gisath020, task  0  0 [25583]: mask 0x1 set
cpu-bind=MASK - gisath018, task  0  0 [14600]: mask 0x4000 set
cpu-bind=MASK - gisath018, task  0  0 [14608]: mask 0x8000 set
cpu-bind=MASK - gisath018, task  0  0 [14616]: mask 0x20 set
cpu-bind=MASK - gisath018, task  0  0 [14624]: mask 0x40 set
cpu-bind=MASK - gisath018, task  0  0 [14631]: mask 0x20000 set
cpu-bind=MASK - gisath018, task  0  0 [14640]: mask 0x10000 set
cpu-bind=MASK - gisath018, task  0  0 [14647]: mask 0x40000 set
cpu-bind=MASK - gisath018, task  0  0 [14657]: mask 0x1 set
cpu-bind=MASK - gisath018, task  0  0 [14665]: mask 0x80000 set
cpu-bind=MASK - gisath018, task  0  0 [14673]: mask 0x100 set
cpu-bind=MASK - gisath018, task  0  0 [14681]: mask 0x400 set
cpu-bind=MASK - gisath018, task  0  0 [14689]: mask 0x1000 set
cpu-bind=MASK - gisath018, task  0  0 [14697]: mask 0x10 set
cpu-bind=MASK - gisath018, task  0  0 [14705]: mask 0x8 set
cpu-bind=MASK - gisath018, task  0  0 [14714]: mask 0x800 set
cpu-bind=MASK - gisath018, task  0  0 [14722]: mask 0x100000 set
cpu-bind=MASK - gisath018, task  0  0 [14730]: mask 0x80 set
cpu-bind=MASK - gisath018, task  0  0 [14738]: mask 0x2000 set
cpu-bind=MASK - gisath018, task  0  0 [14746]: mask 0x200000 set
cpu-bind=MASK - gisath018, task  0  0 [14754]: mask 0x400000 set
cpu-bind=MASK - gisath018, task  0  0 [14762]: mask 0x200 set
cpu-bind=MASK - gisath018, task  0  0 [14770]: mask 0x4000000 set
cpu-bind=MASK - gisath018, task  0  0 [14778]: mask 0x8000000 set
cpu-bind=MASK - gisath018, task  0  0 [14786]: mask 0x1000000 set
cpu-bind=MASK - gisath018, task  0  0 [14792]: mask 0x800000 set
cpu-bind=MASK - gisath018, task  0  0 [14795]: mask 0x2000000 set
srun: Job 677547 step creation temporarily disabled, retrying (Requested nodes are busy)
srun: Job 677547 step creation temporarily disabled, retrying (Requested nodes are busy)
srun: Job 677547 step creation temporarily disabled, retrying (Requested nodes are busy)
srun: Job 677547 step creation temporarily disabled, retrying (Requested nodes are busy)
srun: Job 677547 step creation temporarily disabled, retrying (Requested nodes are busy)
srun: Job 677547 step creation temporarily disabled, retrying (Requested nodes are busy)
srun: Job 677547 step creation temporarily disabled, retrying (Requested nodes are busy)
srun: Job 677547 step creation temporarily disabled, retrying (Requested nodes are busy)
srun: Job 677547 step creation temporarily disabled, retrying (Requested nodes are busy)
srun: Job 677547 step creation temporarily disabled, retrying (Requested nodes are busy)
srun: Job 677547 step creation temporarily disabled, retrying (Requested nodes are busy)

Hi Patrick,

First, to answer your question about this warning:

> srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1

srun requested 2 nodes implicitly - the job allocation is 2 nodes, and srun didn't specify node count, so it uses the number for the job allocation. But, srun requested 1 CPU and 1 task, and you can't run 1 process on 2 nodes. You can specify the number of nodes to silence this warning (-N1).



Now to answer your main question:

There were some changes to srun in 20.11. From our RELEASE_NOTES document:

https://github.com/SchedMD/slurm/blob/slurm-20-11-5-1/RELEASE_NOTES

 -- By default, a step started with srun will be granted exclusive (or non-
    overlapping) access to the resources assigned to that step. No other
    parallel step will be allowed to run on the same resources at the same
    time. This replaces one facet of the '--exclusive' option's behavior, but
    does not imply the '--exact' option described below. To get the previous
    default behavior - which allowed parallel steps to share all resources -
    use the new srun '--overlap' option.
 -- In conjunction to this non-overlapping step allocation behavior being the
    new default, there is an additional new option for step management
    '--exact', which will allow a step access to only those resources requested
    by the step. This is the second half of the '--exclusive' behavior.
    Otherwise, by default all non-gres resources on each node in the allocation
    will be used by the step, making it so no other parallel step will have
    access to those resources unless both steps have specified '--overlap'.

You can find more background about this change here:

https://bugs.schedmd.com/show_bug.cgi?id=10383#c63


In other words, the default behavior for srun is:

* Exclusive access to resources it requests (srun --exclusive)
* All the resources of the job on the node (srun --whole)

These can be overridden by:

* srun --overlap (srun can overlap each other)
* srun --exact (only use exactly the resources requested)


Here's an example: I have 8 cores and 2 threads per core on my nodes.

#!/bin/bash
#SBATCH -n20
set -x
srun --exact -N1 -c1 -n1 whereami
printf "\n\n\n"
srun -N1 -c1 -n1 whereami
printf "\n\n\n"
srun -N2 -n2 whereami | sort


("whereami" is a simple program we wrote that just displays CPU masks. You can also display the masks like you are already doing. I can share this program with you if you want, though.)

$ sbatch 11275.batch 
Submitted batch job 202
$ cat slurm-202.out 
+ srun --exact -N1 -c1 -n1 whereami
0000 n1-1 - Cpus_allowed:       0101    Cpus_allowed_list:      0,8
+ printf '\n\n\n'



+ srun -N1 -c1 -n1 whereami
0000 n1-1 - Cpus_allowed:       1f1f    Cpus_allowed_list:      0-4,8-12
+ printf '\n\n\n'



+ srun -N2 -n2 whereami
+ sort
0000 n1-1 - Cpus_allowed:       1f1f    Cpus_allowed_list:      0-4,8-12
0001 n1-2 - Cpus_allowed:       1f1f    Cpus_allowed_list:      0-4,8-12


My first step uses --exact to get exactly the CPUs it asked for. My second step doesn't use --exact so it is given all the resources on the node. This is what you're seeing. My third step uses all the resources in the job just to show what the CPU bindings are for the entire job across the 2 nodes.

Basically, unless --exact is specified, --cpus-per-task is ignored in a job step.


I admit this was surprising to me, even though I knew about this change to srun - I thought if I specified --cpus-per-task that slurmctld would give me exactly what I asked for. I will look into changing the behavior so that if --cpus-per-task is explicitly requested by the user, then we can imply --exact. But if it turns out that we don't want to do that, then I will at least write a documentation patch to the srun man page to clarify that --cpus-per-task requires --exact.

Can you run your tests with --exact and let me know if it does what you expect?

> If I change the script and add --exclusive to the srun parameters I still get these same "srun: Warning:..."  but the behaviour is back to expected and one core is allocated per task:


This happens because explicitly requesting --exclusive implicitly set --exact.

Hello - thank you for the explanation;
I just did a quick test but I believe I still see some issue when going across nodes. The following script will request two desktops with 3 CPU's configured each:

#!/bin/bash
#SBATCH --ntasks 6
#SBATCH --partition desktop

for i in {1..6}; do
echo $i
srun -N1 -n1 -c1 --exact sleep 150 &
done

wait

SLURM assigns two nodes:
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
           1380409   desktop slurmtes  ngib740  R       2:49      2 giswlx[100-101]

The output file however shows that only 1 CPU is used on the second node (giswlx101):
cpu-bind=MASK - giswlx100, task  0  0 [11655]: mask 0x7 set
1
2
3
4
5
6
cpu-bind=MASK - giswlx100, task  0  0 [11752]: mask 0x1 set
cpu-bind=MASK - giswlx100, task  0  0 [11764]: mask 0x4 set
cpu-bind=MASK - giswlx100, task  0  0 [11767]: mask 0x2 set
cpu-bind=MASK - giswlx101, task  0  0 [25074]: mask 0x1 set
srun: Job 1380409 step creation temporarily disabled, retrying (Requested nodes are busy)
srun: Job 1380409 step creation temporarily disabled, retrying (Requested nodes are busy)

If I use "srun -N1 -n1 -c1 --exact --overlap sleep 150 &", then it is even worse with SLURM starting more tasks (5) on giswlx100 than there are configured CPU's:
cpu-bind=MASK - giswlx100, task  0  0 [16047]: mask 0x7 set
1
2
3
4
5
6
cpu-bind=MASK - giswlx100, task  0  0 [16125]: mask 0x4 set
cpu-bind=MASK - giswlx100, task  0  0 [16138]: mask 0x1 set
cpu-bind=MASK - giswlx100, task  0  0 [16144]: mask 0x1 set
cpu-bind=MASK - giswlx100, task  0  0 [16154]: mask 0x2 set
cpu-bind=MASK - giswlx100, task  0  0 [16155]: mask 0x2 set
cpu-bind=MASK - giswlx101, task  0  0 [26888]: mask 0x1 set

So am I still missing something in the configuration here ?
thanks to advise.

I agree that if cpus-per-tasks is specified it would make sense to imply "exact" or at least issue a warning in the log file that the option is ignored.

(In reply to Patrick from comment #26)
> Hello - thank you for the explanation;
> I just did a quick test but I believe I still see some issue when going
> across nodes. The following script will request two desktops with 3 CPU's
> configured each:
> 
> #!/bin/bash
> #SBATCH --ntasks 6
> #SBATCH --partition desktop
> 
> for i in {1..6}; do
> echo $i
> srun -N1 -n1 -c1 --exact sleep 150 &
> done
> 
> wait

Actually, this job doesn't request two nodes with 3 CPUs each. This job only requests 6 tasks on partition "desktop". The tasks don't have to be distributed evenly on two nodes, and they don't even have to be on two nodes. If Slurm can fit the 6 tasks on one node, then it will try to do that since "block" is the default distribution for tasks across nodes with select/cons_res. You can read more about the distribution in the sbatch/srun/salloc man pages (search for the -m, --distribution option).

To request exactly 3 tasks on 2 nodes, use

sbatch --ntasks-per-node=3 -N2

So it looks like what actually happened is 5 tasks were on one node, and 1 task was on another node.



> If I use "srun -N1 -n1 -c1 --exact --overlap sleep 150 &", then it is even
> worse with SLURM starting more tasks (5) on giswlx100 than there are
> configured CPU's:

--overlap allows steps to share CPUs.

Marshall - the "desktops" are all configured with 3 CPU's:
NodeName=giswlx100 Arch=x86_64 CoresPerSocket=3
   CPUAlloc=0 CPUTot=3 CPULoad=2.01

This means that asking for 6 tasks requests 2 hosts with 3 CPU's each.
The main issue here is that I'm asking for 6 tasks but can only start (srun) 4 at the same time :
(SLURM output file)
1
2
3
4
5
6
cpu-bind=MASK - giswlx100, task  0  0 [11752]: mask 0x1 set
cpu-bind=MASK - giswlx100, task  0  0 [11764]: mask 0x4 set
cpu-bind=MASK - giswlx100, task  0  0 [11767]: mask 0x2 set
cpu-bind=MASK - giswlx101, task  0  0 [25074]: mask 0x1 set
srun: Job 1380409 step creation temporarily disabled, retrying (Requested nodes are busy)
srun: Job 1380409 step creation temporarily disabled, retrying (Requested nodes are busy)

To illustrate this better here's a test on 3 nodes:

#!/bin/bash
#SBATCH --ntasks 9
#SBATCH --ntasks-per-node 3
#SBATCH --partition desktop

for i in {1..9}; do
echo $i
srun -N1 -n1 -c1 --exact sleep 30 &
done

wait

SLURM will use 3 CPU's / tasks on the first assigned host (giswlx100) but only a single task on each of the 2 other hosts:

cpu-bind=MASK - giswlx100, task  0  0 [32493]: mask 0x1 set
cpu-bind=MASK - giswlx100, task  0  0 [32498]: mask 0x4 set
cpu-bind=MASK - giswlx100, task  0  0 [32504]: mask 0x2 set
cpu-bind=MASK - giswlx101, task  0  0 [13746]: mask 0x1 set
cpu-bind=MASK - giswlx102, task  0  0 [27852]: mask 0x1 set
srun: Job 1388346 step creation temporarily disabled, retrying (Requested nodes are busy)
srun: Job 1388346 step creation temporarily disabled, retrying (Requested nodes are busy)
srun: Step created for job 1388346
srun: Job 1388346 step creation temporarily disabled, retrying (Requested nodes are busy)
srun: Job 1388346 step creation temporarily disabled, retrying (Requested nodes are busy)
srun: Job 1388346 step creation still disabled, retrying (Requested nodes are busy)
srun: Step created for job 1388346
srun: Job 1388346 step creation still disabled, retrying (Requested nodes are busy)
srun: Step created for job 1388346
srun: Job 1388346 step creation still disabled, retrying (Requested nodes are busy)
srun: Step created for job 1388346
cpu-bind=MASK - giswlx100, task  0  0 [321]: mask 0x1 set
cpu-bind=MASK - giswlx100, task  0  0 [338]: mask 0x2 set
cpu-bind=MASK - giswlx102, task  0  0 [28906]: mask 0x1 set
cpu-bind=MASK - giswlx101, task  0  0 [14921]: mask 0x1 set

Patrick,

Thanks for the clarification. I can reproduce what you're seeing in your latest comment, and I agree it seems like a bug. But it's different from the original issue you reported about --cpus-per-task, so I've created bug 11357 to handle this and added you to CC. Let's continue our conversation over there, and we'll leave this bug to handle --cpus-per-task implying --exact.

Patrick,

We've pushed a fix for --cpus-per-task (and --threads-per-core, which had the same issue) to imply --exact. We pushed this to the master branch since it is a change in behavior, and documented the behavior change in NEWS, RELEASE_NOTES, and the srun man page. So this will be in 21.08 when it is released (this August).

If you want this for 20.11, you should be able to cherry pick the patch (just src/srun/libsrun/launch.c, since the changes to other files are only documentation changes).

Closing this as resolved/fixed in 21.08.

I realized I never made the commit hash public. Here's the commit with the change:

https://github.com/SchedMD/slurm/commit/e01e884f3c294


(Sorry about the extra email.)

Hi Patrick, I just wanted to give you some updates on this:

It turned out that due to a bug, -c/--cpus-per-task and --threads-per-core did NOT imply --exact in 21.08.0 through 21.08.4. In 21.08.5, we fixed that bug so that they imply --exact properly. However, we discovered that this broke MPI programs pretty badly, since mpirun can't work like it needs to when --exact is specified. So in 21.08.6+, we reverted -c/--cpus-per-task and --threads-per-core implying --exact.

However, in 22.05, we are going to make -c/--cpus-per-task imply --exact again, but we are also going to change it so that srun does NOT inherit any -c specified by salloc/sbatch. This will give us the best of both worlds - it won't break MPI programs, and it also will fix the issue you highlighted in this ticket (where `srun -cX ...` gives you the job's whole allocation when it really doesn't make sense for it to).

What this means for you is that in 21.08.6+, make sure users specify --exact when doing `srun -c ...` to get only the CPUs you expect. In 22.05, you can get rid of the extra --exact.

See bug 13351 comment 76 for more details.

Thanks!
-Michael

thank you for the update Michal, we'll keep that in mind when we will update to the latest SLURM release.