Ticket 1606

Hey Brian,

I'll try to reproduce it with your slurm.conf and get back to you. 

Thanks,
Brian

I'm able to reproduce the problem and understand what's happening. Slurm is trying to get the 20 nodes and the 400 tasks from the 24cpu,12core nodes but can't. Requesting the feature filters out the 24cpu nodes. 

A couple of workarounds are:
1. specify the feature for these types of jobs.
2. specify --ntasks-per-node=20
3. change the node's weights so that the 40cpu nodes are looked at first.

The cons_res plugin doesn't have this issue. I will look into possibly changing the select/linear plugin to account for this type of situation.

Let me know if you have any questions.

Thanks,
Brian

So I had another question based on your response.  You point out that select/cons_res does not suffer from this.  If I am not focused on sharing nodes, nor focused on allocating less then a whole node should I consider moving to select/cons_res?

I guess the simpler question here is "if" nobody really uses select/linear and select/cons_res is much more heavily used, tested, and developed maybe we should change?

I'd rather not change, but I'd also rather pick the path that would give us the best chance for being in the "better" state.  Guide me on this please.  If linear is well used/developed just tell me I'm good. :)  Thanks.

(In reply to Brian Haymore from comment #3)
> I guess the simpler question here is "if" nobody really uses select/linear
> and select/cons_res is much more heavily used, tested, and developed

Yes on all counts.

> maybe
> we should change?

Your call. You can certainly use select/cons_res to allocate whole nodes, and it is much more heavily used, tested, and developed...

So I'm playing with this on a test cluster of 8 8 core nodes.  The setup I have uses:

SelectType=select/cons_res
SelectTypeParameters=CR_Core,CR_ONE_TASK_PER_CORE

From reading things and simple tests this seems to give me pretty much what I had with linear.

However from reading things in this mode the default suggests that if I were to ask for a job to run on 6 nodes and have a ntasks of 36 that it would evenly put 6 processes per node, yet it is not.  I'm finding the allocation of cores on nodes to be a bit odd.  For example doing what I said above, -N 6 -n 36 I get this distribution:

SLURM_STEP_TASKS_PER_NODE=7(x4),6,2
SLURM_STEP_NODELIST=sp[003-008]


Just for reference my node definitions are as below.  You will see that all nodes have the same socket, core, thread numbers...

# CHPC General Nodes
NodeName=sp[001-002] Feature=chpc,c8 Weight=1 NodeAddr=sp[001-002] Sockets=2 CPUs=16 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=24148 TmpDisk=103373
#
# Cardoen Owner Nodes
NodeName=sp[003-004] Feature=cardoen,c8 Weight=1 NodeAddr=sp[003-004] Sockets=2 CPUs=16 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=24148 TmpDisk=103373
#
# Cuma Onwer Nodes
NodeName=sp005 Feature=cuma,c8 Weight=1 NodeAddr=sp005 Sockets=2 CPUs=16 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=24148 TmpDisk=103373
#
# Orendt Owner Nodes
NodeName=sp[006-008] Feature=orendt,c8 Weight=1 NodeAddr=sp[006-008] Sockets=2 CPUs=16 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=24148 TmpDisk=103373

So why am I getting allocated 7 cores on the first 4 nodes, 6 on the 5th node and 2 on the 6th node?

There's one more bit that you need. If you want to continue allocating whole nodes to jobs, add "Shared=exclusive" to the partition(s) configuration line. The tasks will then be evenly distributed across those nodes.

Without that, your job will get 36 cores (one for each task) distributed in an arbitrary fashion across those 6 nodes.

Thanks!  That got it.  Though the web docs seem to suggest that this should have been the behavior from the start when you read through the SelectTypeParameters for cons_res.  Specifically the wording from the slurm.conf documentation and the CR_Pack_Nodes implies that the without CR_Pack_Nodes set that job tasks will be evenly distributed, but with they will be packed in.  There isn't any mention onf this on the cons_res specific documentation page either.  Maybe an update to the docs could get into the queue for when time allows?  Thanks again for the help!

(In reply to Brian Haymore from comment #7)
> Thanks!  That got it
> Maybe an update to the docs could get
> into the queue for when time allows?

Done in v14.11.6 (when released):
https://github.com/SchedMD/slurm/commit/a001b4b67aa4986ddf3664f59b455e7604225015