Currently if you provide a maximum node count with -N that's smaller than the number of nodes in the -w nodelist, srun simply truncates the node list to the maximum node count. For example, "srun -N 1-1 -w nid000[12-15,36-39,44-47]" will only place on nid00012. It would be nice if the job could be placed on any of those nodes, so if nid 12 is busy the job can still run.
The job or step allocation is intended to include ALL nodes in the hostlist and possibly additional resources as needed. There is no mechanism available today to select one node from an arbitrary node list. Available option today would be to either specify some node feature that makes these nodes more desirable to use (--constraint=haswell) or explicitly exclude all other nodes (--exclude=tux[0-100,110-1023]). $ srun -N1-1 -w tux2,tux3 date srun: error: Required nodelist includes more nodes than permitted by max-node count (2 > 1). Eliminating nodes from the nodelist.
Our internal users wanted a way for -w to work more like the ALPS/PBS/Moab node list, because they use it heavily with cnselect. I'm trying to get them to think in more of a Slurm fashion. The admin had trouble configuring features on his first try so I'm working on setting up features on another system to give him an example.
Our internal admin was able to configure features on the second try. I think they would still like this RFE however.
It's definitely a significant effort (new field in multiple data structures plus matching logic), so it's not likely to happen for a while.
Is there interest on the Cray side to have us scope out a custom development engagement for this request?
Is there still interested in this?
There is still interest but by now most people here who ran into it have worked around it.
Our users have interest in this as well; but the behavior I'm seeing is different than the behavior described. I expect that it's been modified to just fail faster, rather than implement the requested feature: $ sbatch --qos blanca-ics -a 1-150%50 -w bnode0103,bnode0104 -N 1 ~/btest-array.sh sbatch: error: Batch job submission failed: Node count specification invalid To be clear, I'd want this to run jobs on either bnode0103 or bnode0104, possibly preferring bnode0103 until its resources are consumed (though a round-robin balancing would probably be better).
I should also say that the --nodelist entry in the manpage sbatch seems to explicitly state that this *should* work: > Unless constrained by the maximum node count, the job will contain all of these hosts. What does "unless constrained by the maximum node count" mean, if it doesn't mean that you can constrain the maximum node count with -N? ~jonathon (In reply to Jonathon Anderson from comment #8) > Our users have interest in this as well; but the behavior I'm seeing is > different than the behavior described. I expect that it's been modified to > just fail faster, rather than implement the requested feature: > > $ sbatch --qos blanca-ics -a 1-150%50 -w bnode0103,bnode0104 -N 1 > ~/btest-array.sh > sbatch: error: Batch job submission failed: Node count specification invalid > > To be clear, I'd want this to run jobs on either bnode0103 or bnode0104, > possibly preferring bnode0103 until its resources are consumed (though a > round-robin balancing would probably be better).
(In reply to Jonathon Anderson from comment #9) > I should also say that the --nodelist entry in the manpage sbatch seems to > explicitly state that this *should* work: > > > Unless constrained by the maximum node count, the job will contain all of these hosts. > > What does "unless constrained by the maximum node count" mean, if it doesn't > mean that you can constrain the maximum node count with -N? > > ~jonathon I've modified the documentation to match the way the logic has always worked. See commit: https://github.com/SchedMD/slurm/commit/9faadae0265b5baa19d97d197547f2888b64f697 -Unless constrained by the maximum node count, the job will contain -\fIall\fR of these hosts. +The job will contain \fIall\fR of these hosts and possibly additional hosts +as needed to satisfy resource requirements.
(In reply to Jonathon Anderson from comment #8) > Our users have interest in this as well; but the behavior I'm seeing is > different than the behavior described. I expect that it's been modified to > just fail faster, rather than implement the requested feature: > > $ sbatch --qos blanca-ics -a 1-150%50 -w bnode0103,bnode0104 -N 1 > ~/btest-array.sh > sbatch: error: Batch job submission failed: Node count specification invalid > > To be clear, I'd want this to run jobs on either bnode0103 or bnode0104, > possibly preferring bnode0103 until its resources are consumed (though a > round-robin balancing would probably be better). The request above is interpreted by Slurm as saying, allocate this job exactly one node and include the nodes bnode0103 and bnode0104, which does not compute. There are two options to address this: 1. Identify some feature of nodes bnode0103 and bnode0104 add add it to the configuration and job request. For example, in slurm.conf NodeName=bnode0103,bnode0104 Feature=haswell ... Then in the job submission: sbatch --qos blanca-ics -a 1-150%50 -C haswell -N 1 ~/btest-array.sh 2. List all of the nodes which you do NOT want to use with the -x option.
What would it take to get the feature that was previously documented ("unless constrained by the maximum node count") implemented? I appreciate reconciling the documentation with what the code actually does; but it'd be better if the code was changed to actually perform the feature that was previously documented. ~jonathon (In reply to Moe Jette from comment #10) > (In reply to Jonathon Anderson from comment #9) > > I should also say that the --nodelist entry in the manpage sbatch seems to > > explicitly state that this *should* work: > > > > > Unless constrained by the maximum node count, the job will contain all of these hosts. > > > > What does "unless constrained by the maximum node count" mean, if it doesn't > > mean that you can constrain the maximum node count with -N? > > > > ~jonathon > > I've modified the documentation to match the way the logic has always > worked. See commit: > https://github.com/SchedMD/slurm/commit/ > 9faadae0265b5baa19d97d197547f2888b64f697 > > -Unless constrained by the maximum node count, the job will contain > -\fIall\fR of these hosts. > +The job will contain \fIall\fR of these hosts and possibly additional hosts > +as needed to satisfy resource requirements.
I understand how slurm is interpreting this; but I also think it's pretty clear what the *correct* interpretation is. Maybe it will be more palatable if, in stead of requesting a change to the implementation of --nodelist, we ask for an --include that is the opposite of --exclude? Cause that's what we actually want, and what we expected --nodelist to do in its absense. ~jonathon (In reply to Moe Jette from comment #11) > (In reply to Jonathon Anderson from comment #8) > > Our users have interest in this as well; but the behavior I'm seeing is > > different than the behavior described. I expect that it's been modified to > > just fail faster, rather than implement the requested feature: > > > > $ sbatch --qos blanca-ics -a 1-150%50 -w bnode0103,bnode0104 -N 1 > > ~/btest-array.sh > > sbatch: error: Batch job submission failed: Node count specification invalid > > > > To be clear, I'd want this to run jobs on either bnode0103 or bnode0104, > > possibly preferring bnode0103 until its resources are consumed (though a > > round-robin balancing would probably be better). > > The request above is interpreted by Slurm as saying, allocate this job > exactly one node and include the nodes bnode0103 and bnode0104, which does > not compute. > > There are two options to address this: > 1. Identify some feature of nodes bnode0103 and bnode0104 add add it to the > configuration and job request. For example, in slurm.conf > NodeName=bnode0103,bnode0104 Feature=haswell ... > Then in the job submission: > sbatch --qos blanca-ics -a 1-150%50 -C haswell -N 1 ~/btest-array.sh > > 2. List all of the nodes which you do NOT want to use with the -x option.
(In reply to Jonathon Anderson from comment #12) > What would it take to get the feature that was previously documented > ("unless constrained by the maximum node count") implemented? I appreciate > reconciling the documentation with what the code actually does; but it'd be > better if the code was changed to actually perform the feature that was > previously documented. Note that the current slurm logic supports a command line of this sort sbatch --nodelist=tux12 -N20 ... (i.e. give me 20 nodes and node tux12 must be one of them) or sbatch --nodelist=tux12 -n5000 -c1 ... (i.e. give me 5000 CPUs and node tux12 must be one of the nodes), which we don't want to break. Probably the easiest thing to do would be to write a job_submit plugin that will note the job request specified 2 specific nodes to include in the allocation, but the job only wants one node, so translate the --nodelist option to a --exclude option with every other node. This would not address the case where a node count is not specified, but rather task count, socket count, tasks per baseboard, etc. sbatch --nodelist=tux[12-20] -n1 -c1 ... (but no -N1) OR sbatch --nodelist=tux[12-20] -n2 --ntasks-pernode=2 ...
I've been trying to train our users/admins to use node features for this kind of thing. It fits all of the use cases I've found so far.
(In reply to David Gloe from comment #15) > I've been trying to train our users/admins to use node features for this > kind of thing. It fits all of the use cases I've found so far. I agree, that is the best way to go. Jonathon, What's special about nodes bnode0103 and bnode0104?
(In reply to Moe Jette from comment #16) > Jonathon, What's special about nodes bnode0103 and bnode0104? Our users continue to do The Wrong Thing.™ For some reason, they don't trust the scheduler to schedule jobs in their condo nodes between users; so they whiteboard-schedule access to specific nodes per-user, and then run all their jobs with --nodelist. :/ This works fine for them in the one-node-per-user case; but is breaking for them in this case where they want to distribute an array of serial jobs across a set of two specific nodes. I don't mean to pretend like this is correct--in fact, I find it horrifying--but I can still argue that it should be *possible*. Especially given the existence of --exclude, it seems strange that there's no --include, or equivalent functionality in --nodelist.
(In reply to Jonathon Anderson from comment #17) > (In reply to Moe Jette from comment #16) > > Jonathon, What's special about nodes bnode0103 and bnode0104? > > Our users continue to do The Wrong Thing.™ For some reason, they don't trust > the scheduler to schedule jobs in their condo nodes between users; so they > whiteboard-schedule access to specific nodes per-user, and then run all > their jobs with --nodelist. :/ This works fine for them in the > one-node-per-user case; but is breaking for them in this case where they > want to distribute an array of serial jobs across a set of two specific > nodes. > > I don't mean to pretend like this is correct--in fact, I find it > horrifying--but I can still argue that it should be *possible*. Especially > given the existence of --exclude, it seems strange that there's no > --include, or equivalent functionality in --nodelist. That seems like a really bad way to go. Carving up the machine like this will definitely adversely effect utilization and responsiveness. It would be best to use a minimum number of partitions/queues and use fair-share scheduling to apportion resources, but I'm probably preaching to the choir. If you really want to carve up the machine on a node-by-node basis, advanced reservations might be a good option. See: http://slurm.schedmd.com/reservations.html The reason for the --exclude functionality is so that users can exclude nodes that they believe might be failing. The reason for --nodelist is so that users can stage-in files with one job, then run the compute portion in another job allocation that includes the node with the staged-in files.
