Ticket 2520

Summary: srun exclusive trouble
Product: Slurm Reporter: Josko Plazonic <plazonic>
Component: slurmdAssignee: Danny Auble <dauble>
Status: RESOLVED FIXED QA Contact:
Severity: 4 - Minor Issue    
Priority: ---    
Version: 15.08.8   
Hardware: Linux   
OS: Linux   
Site: Princeton (PICSciE) Slinky Site: ---
Alineos Sites: --- Atos/Eviden Sites: ---
Confidential Site: --- Coreweave sites: ---
Cray Sites: --- DS9 clusters: ---
Google sites: --- HPCnow Sites: ---
HPE Sites: --- IBM Sites: ---
NOAA SIte: --- NoveTech Sites: ---
Nvidia HWinf-CS Sites: --- OCF Sites: ---
Recursion Pharma Sites: --- SFW Sites: ---
SNIC sites: --- Tzag Elita Sites: ---
Linux Distro: --- Machine Name:
CLE Version: Version Fixed: 16.05.0-rc2
Target Release: --- DevPrio: ---
Emory-Cloud Sites: ---
Attachments: Tiger slurm.conf

Description Josko Plazonic 2016-03-06 23:36:04 MST 
Hullo,

jobs setup like:

#!/bin/bash
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=16 
#SBATCH --mem-per-cpu=1000 
#SBATCH -t 30:00

srun -n 20 --exclusive /home/plazonic/a.sh 300 &

srun -n 40 --exclusive /home/plazonic/a.sh 301 &
wait

used to run but we are not sure when it stopped (I am pretty sure it does not run with 15.08.04 either). But:

#!/bin/bash
#SBATCH -n 60
#SBATCH --nodes=4
#SBATCH --mem-per-cpu=1000 
#SBATCH -t 30:00

does run (and distributes jobs fairly across nodes - 5 each for first, 10 each for second). So we need a bit of a clarification here - we want two different tasks to run concurrently and not share CPUs.  We know this used to run with first allocation (probably in 14.11.* time), why doesn't it run now?

A related question is that I don't seem to be able to achieve/control packed allocation, at least not in the way I'd expect.  I tried adding -m block,Pack but does not pack the way I'd expect (16 first node, 4 next for first step and similar for 2nd) and frequently just goes with "fair" (5 each, 10 each).
Comment 1 Tim Wickberg 2016-03-07 08:52:07 MST 
I'd asked on 2521, but just for posterity - the slurm.conf file would help tremendously. I'm guessing you have nodes with 2 sockets, 8 cores, 2 threads, and CR_ONE_TASK_PER_NODE set, but it'd be better if I definitely knew that before trying to reproduce.

Some of this has may have been fixed already in 16.05, but I want to narrow down the issues here, and make sure all of this has been addressed, and give you some potential workarounds if appropriate for 15.08. The task allocation and placement is some of the trickiest logic within Slurm, and as you've seen here there can be some odd behavior from certain combinations of options.

- Tim
Comment 2 Josko Plazonic 2016-03-07 09:22:31 MST 
Created attachment 2836 [details]
Tiger slurm.conf
Comment 3 Tim Wickberg 2016-03-08 03:41:57 MST 
It looks like this is about memory, task placement seems to be working correctly here.

As best I can tell, the first 'srun' is being allocated all of the memory in each node, and the second srun is blocking waiting for memory to free up before it proceeds. This is a side effect of CR_Core_Memory, I'm guessing you may not have had that enabled before, and/or Slurm's enforcement of that has changed.

Can you re-run your tests with --mem set for each srun and see if things behave as you expect?
Comment 4 Josko Plazonic 2016-03-08 09:56:13 MST 
Hm, did not help:

#!/bin/bash
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=16 
#SBATCH --mem-per-cpu=1000 
#SBATCH -t 10:00

srun --mem=1000 -n 20 --exclusive /home/plazonic/a.sh 300 &
srun --mem=1000 -n 40 --exclusive /home/plazonic/a.sh 301 &
wait

did not run - I also tried --mem-per-cpu=1000 on each srun and --mem-per-cpu=900.  Runs first one and second one sits and waits.

Can you reproduce this in 15.08.08? If yes is it fixed in 16.05? We have a workaround (i.e. specifying -n rather then --nodes) so we can live with it if we knew that at some point in near future it will be fixed in 16.05...
Comment 5 Danny Auble 2016-03-18 10:05:33 MDT 
Josko, I can reproduce the exact situation situation fairly easy.

I have gone back to 14.11.7 and can't find a place where the behavior is different though.

I'll see if I can look at this next week, but the --ntasks-per-node definitely seems to be the issue as you point out.  If you don't use it things work just as you would expect.  I would suggest using -n instead if Node's isn't important you can just drop that as well and just have -n.
Comment 6 Josko Plazonic 2016-03-21 00:14:37 MDT 
Mornin',

I assumed it was working with 14.11.* as the user claimed it used to work and now it doesn't.  Can't explain it (he could be wrong but unlikely, must have worked at some point), weird.

We are aware of the workaround which is why I tagged it as a minor issue, it would just be good to get it fixed at some point, if possible...  So no hurry.

Thanks!
Comment 7 Danny Auble 2016-05-11 09:00:28 MDT 
Josko, I finally got back to this today.

It turns out the problem is fairly simple to explain (at least understand what is happening).

You can translate your script into this...


#!/bin/bash
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=16 
#SBATCH --mem-per-cpu=1000 
#SBATCH -t 10:00

srun -N4 --mem=1000 -n 20 --ntasks-per-node=16 --exclusive /home/plazonic/a.sh 300 &
srun -N4 --mem=1000 -n 40 --ntasks-per-node=16 --exclusive /home/plazonic/a.sh 301 &
wait

Meaning the the env var SLURM_NTASKS_PER_NODE is set on the allocation which will then get read in by each srun causing conflicting info.

There are a few ways around this outside of what we have discussed.

- Use the --exclusive option on the allocation instead of the --ntasks-per-node (this will get you the whole node but it looks like that is what you want anyway)
- Unset the SLURM_NTASKS_PER_NODE env var before you call the srun lines.
- Explicitly set the --ntasks-per-node to something that makes since.
- Apply commit 72ed146cd2a6f and ignore the warning. (Built on 16.05, but I am expecting it will go in 15.08 cleanly)

As the last option notes in 16.05 I am printing a warning about the bad math and ignoring the ntasks-per-node option when nodes and tasks are specified along with ntasks-per-node and the math doesn't add up.

If you have a better idea please let me know and reopen the bug.  At least this will make it clear as to what is happening and won't (hopefully) cause as much confusion.
Comment 8 Josko Plazonic 2016-05-12 05:24:03 MDT 
When you put it that way how could I not understand it - fantastic explanation..  In other words - any setting sbatch gets srun inherits and all trouble descends from it.  

Solution is perfect too - thanks!  And no trouble for taking this long - we've seen this "issue" quite rarely and had a workaround.

Thanks!
Comment 9 Danny Auble 2016-05-12 05:34:50 MDT 
Yeah, most options set in sbatch are inherited by srun unless it over rides it.  Glad you understood :).