Ticket 3013

Summary: SC15 Configuration Suggestions
Product: Slurm Reporter: Stephen Fralich <sjf4>
Component: ConfigurationAssignee: Tim Wickberg <tim>
Status: RESOLVED INFOGIVEN QA Contact:
Severity: 4 - Minor Issue    
Priority: ---    
Version: 16.05.4   
Hardware: Linux   
OS: Linux   
Site: University of Washington Slinky Site: ---
Alineos Sites: --- Atos/Eviden Sites: ---
Confidential Site: --- Coreweave sites: ---
Cray Sites: --- DS9 clusters: ---
Google sites: --- HPCnow Sites: ---
HPE Sites: --- IBM Sites: ---
NOAA SIte: --- NoveTech Sites: ---
Nvidia HWinf-CS Sites: --- OCF Sites: ---
Recursion Pharma Sites: --- SFW Sites: ---
SNIC sites: --- Tzag Elita Sites: ---
Linux Distro: --- Machine Name:
CLE Version: Version Fixed:
Target Release: --- DevPrio: ---
Emory-Cloud Sites: ---
Attachments: Current test configuration

Description Stephen Fralich 2016-08-22 13:34:47 MDT
Created attachment 3428 [details]
Current test configuration

At Supercomputing last year you all suggested we make pools of common hardware nodes  allowing a group to use up to a certain number rather than assign specific node to specific groups. I'm interested in adopting that model. 

I've attached the test configuration I've been using previously. It implements the system I originally planned on before our conversation at Supercomputing.

Here's the text of a description of our requirements I've previously sent in e-email before as well:
We have a condo cluster, so specific groups own specific nodes.

Node owners jobs run on their own nodes by default.

Other users can run on other group's nodes via a preemptee queue, but
must be preempted within about a minute if the owner needs the
resource

A user's priority in the preemptee queue is determined by the number
of nodes his/her group owns and group usage

Optional usage based priority on some group's nodes

We have a preference for per node scheduling rather than per core.
Being able to designate some nodes as optionally per-core scheduled if
jobs are coming from a given queue is important. So in other words,
when the node is running a preemptee job, we want it per node
scheduled, but for some nodes, while the owner is using the node we
want it per-core scheduled.

Our system is heterogeneous: we have nodes with 8, 12, 16, and 32
cores. It's useful for users to be able to request 1 or more nodes
with any number of cores, receive a set of nodes with the same core
count, and have the accounting reflect they used the entire node.

Partition support to reflect network topology

CPU hour and job count accounting by group, queue, and user. Some
interest in accelerator usage tracking as well.

Provide a delay for preempted jobs to checkpoint at the application
level rather than just be kill -KILL'ed immediately.

Limit number of jobs per user that can accumulate queue time based priority
Comment 1 Tim Wickberg 2016-08-23 08:08:42 MDT
Moving to Sev4; please always submit new issues as Sev1-4. Sev5 is longer-term feature requests and won't get updated as promptly.

I'm working on a more in-depth response to this. But one thing to point you at is the QOS documentation page which describes the PartitionQOS feature that could be used to implement "floating" partitions, where each has a set resource limit but is allowed access to any of the underlying hardware.

http://slurm.schedmd.com/qos.html

(In reply to Stephen Fralich from comment #0)
> Created attachment 3428 [details]
> Current test configuration
> 
> At Supercomputing last year you all suggested we make pools of common
> hardware nodes  allowing a group to use up to a certain number rather than
> assign specific node to specific groups. I'm interested in adopting that
> model. 
> 
> I've attached the test configuration I've been using previously. It
> implements the system I originally planned on before our conversation at
> Supercomputing.
> 
> Here's the text of a description of our requirements I've previously sent in
> e-email before as well:
> We have a condo cluster, so specific groups own specific nodes.
> 
> Node owners jobs run on their own nodes by default.
> 
> Other users can run on other group's nodes via a preemptee queue, but
> must be preempted within about a minute if the owner needs the
> resource
> 
> A user's priority in the preemptee queue is determined by the number
> of nodes his/her group owns and group usage
> 
> Optional usage based priority on some group's nodes
> 
> We have a preference for per node scheduling rather than per core.
> Being able to designate some nodes as optionally per-core scheduled if
> jobs are coming from a given queue is important. So in other words,
> when the node is running a preemptee job, we want it per node
> scheduled, but for some nodes, while the owner is using the node we
> want it per-core scheduled.
> 
> Our system is heterogeneous: we have nodes with 8, 12, 16, and 32
> cores. It's useful for users to be able to request 1 or more nodes
> with any number of cores, receive a set of nodes with the same core
> count, and have the accounting reflect they used the entire node.
> 
> Partition support to reflect network topology
> 
> CPU hour and job count accounting by group, queue, and user. Some
> interest in accelerator usage tracking as well.
> 
> Provide a delay for preempted jobs to checkpoint at the application
> level rather than just be kill -KILL'ed immediately.
> 
> Limit number of jobs per user that can accumulate queue time based priority
Comment 2 Tim Wickberg 2016-08-23 09:36:41 MDT
My responses are inline below interspersed with some configuration values to consider adding to slurm.conf.

Please let me know if I can elaborate further on any of these.

- Tim

(In reply to Stephen Fralich from comment #0)
> Created attachment 3428 [details]
> Current test configuration
> 
> At Supercomputing last year you all suggested we make pools of common
> hardware nodes  allowing a group to use up to a certain number rather than
> assign specific node to specific groups. I'm interested in adopting that
> model. 

The 'floating' partitions can be implemented with what is called a PartitionQOS.

The QOS describes the relevant resource limits for that partition, then Slurm
will keep track and only allow that partition to access up to that amount of
resources in the nodes that are made available to it.

http://slurm.schedmd.com/qos.html

> I've attached the test configuration I've been using previously. It
> implements the system I originally planned on before our conversation at
> Supercomputing.
> 
> Here's the text of a description of our requirements I've previously sent in
> e-email before as well:
> We have a condo cluster, so specific groups own specific nodes.
> 
> Node owners jobs run on their own nodes by default.
> 
> Other users can run on other group's nodes via a preemptee queue, but
> must be preempted within about a minute if the owner needs the
> resource

http://slurm.schedmd.com/preempt.html

It looks like you already have this in place.

> A user's priority in the preemptee queue is determined by the number
> of nodes his/her group owns and group usage

Priority between users is best established through FairShare scheduling. 
You'd want to structure the fairshare values between the different accounts
in sacctmgr appropriately.

I would recommend using the Fair_Tree algorithm developed by BYU in place of
traditional FairShare. http://slurm.schedmd.com/fair_tree.html

(Just set PriorityFlags=FAIR_TREE)

> Optional usage based priority on some group's nodes

Fairshare will still sort between users in the account on all partitions,
although this is obviously only going to impact users in that account.

One caveat: usage under a group's own nodes would accumulate and be factored
in the global fair share calculation. If you'd prefer that a group's usage on
their own nodes be disregarded, you can set

TRESBillingWeights=CPU=0,Mem=0,GRES/gpu=0 on those partitions.

> We have a preference for per node scheduling rather than per core.
> Being able to designate some nodes as optionally per-core scheduled if
> jobs are coming from a given queue is important. So in other words,
> when the node is running a preemptee job, we want it per node
> scheduled, but for some nodes, while the owner is using the node we
> want it per-core scheduled.

Setting OverSubscribe=Exclusive on the partition would force that partition to
only schedule full nodes. It looks like you already have this in effect.
('Shared' was renamed to 'OverSubscribe' in the 16.05 release although both
terms still function.)

> Our system is heterogeneous: we have nodes with 8, 12, 16, and 32
> cores. It's useful for users to be able to request 1 or more nodes
> with any number of cores, receive a set of nodes with the same core
> count, and have the accounting reflect they used the entire node.

Accounting should reflect the full node worth of CPUs. Accounting is always by
CPU*minute.

> Partition support to reflect network topology
>
> CPU hour and job count accounting by group, queue, and user. Some
> interest in accelerator usage tracking as well.

For accelerator tracking, you'll want to add in the GPU Gres to the relevant nodes.

GresTypes=gpu
AccountingStorageTRES=gres/gpu

> Provide a delay for preempted jobs to checkpoint at the application
> level rather than just be kill -KILL'ed immediately.

I see you have set KillWait=30 which should provide for this.

> Limit number of jobs per user that can accumulate queue time based priority

There's no native way to do this, but a few alternatives:

- Set the various PriorityWeight values to favor other values (usually 
Fairshare) more than Age.

- Limit the number of jobs that user can submit to that partition using a PartitionQOS.
Comment 3 Stephen Fralich 2016-08-25 14:38:07 MDT
I have a question about floating paritions. Let say:
I have 10 nodes with 128GB and 15 nodes with 256GB

groupA owns 3 128GB nodes and 2 256GB nodes
groupB owns 7 128GB nodes 13 256GB nodes

Is there a way for groupA to run a single job using 5 nodes? It seems
like the nodes with different memory configurations would have to be
in different partitions.

Let me know if I'm misunderstanding. Thanks,

Stephen


On Tue, Aug 23, 2016 at 8:36 AM,  <bugs@schedmd.com> wrote:
> Comment # 2 on bug 3013 from Tim Wickberg
>
> My responses are inline below interspersed with some configuration values to
> consider adding to slurm.conf.
>
> Please let me know if I can elaborate further on any of these.
>
> - Tim
>
> (In reply to Stephen Fralich from comment #0)
>> Created attachment 3428 [details]
>> Current test configuration
>>
>> At Supercomputing last year you all suggested we make pools of common
>> hardware nodes  allowing a group to use up to a certain number rather than
>> assign specific node to specific groups. I'm interested in adopting that
>> model.
>
> The 'floating' partitions can be implemented with what is called a
> PartitionQOS.
>
> The QOS describes the relevant resource limits for that partition, then
> Slurm
> will keep track and only allow that partition to access up to that amount of
> resources in the nodes that are made available to it.
>
> http://slurm.schedmd.com/qos.html
>
>> I've attached the test configuration I've been using previously. It
>> implements the system I originally planned on before our conversation at
>> Supercomputing.
>>
>> Here's the text of a description of our requirements I've previously sent
>> in
>> e-email before as well:
>> We have a condo cluster, so specific groups own specific nodes.
>>
>> Node owners jobs run on their own nodes by default.
>>
>> Other users can run on other group's nodes via a preemptee queue, but
>> must be preempted within about a minute if the owner needs the
>> resource
>
> http://slurm.schedmd.com/preempt.html
>
> It looks like you already have this in place.
>
>> A user's priority in the preemptee queue is determined by the number
>> of nodes his/her group owns and group usage
>
> Priority between users is best established through FairShare scheduling.
> You'd want to structure the fairshare values between the different accounts
> in sacctmgr appropriately.
>
> I would recommend using the Fair_Tree algorithm developed by BYU in place of
> traditional FairShare. http://slurm.schedmd.com/fair_tree.html
>
> (Just set PriorityFlags=FAIR_TREE)
>
>> Optional usage based priority on some group's nodes
>
> Fairshare will still sort between users in the account on all partitions,
> although this is obviously only going to impact users in that account.
>
> One caveat: usage under a group's own nodes would accumulate and be factored
> in the global fair share calculation. If you'd prefer that a group's usage
> on
> their own nodes be disregarded, you can set
>
> TRESBillingWeights=CPU=0,Mem=0,GRES/gpu=0 on those partitions.
>
>> We have a preference for per node scheduling rather than per core.
>> Being able to designate some nodes as optionally per-core scheduled if
>> jobs are coming from a given queue is important. So in other words,
>> when the node is running a preemptee job, we want it per node
>> scheduled, but for some nodes, while the owner is using the node we
>> want it per-core scheduled.
>
> Setting OverSubscribe=Exclusive on the partition would force that partition
> to
> only schedule full nodes. It looks like you already have this in effect.
> ('Shared' was renamed to 'OverSubscribe' in the 16.05 release although both
> terms still function.)
>
>> Our system is heterogeneous: we have nodes with 8, 12, 16, and 32
>> cores. It's useful for users to be able to request 1 or more nodes
>> with any number of cores, receive a set of nodes with the same core
>> count, and have the accounting reflect they used the entire node.
>
> Accounting should reflect the full node worth of CPUs. Accounting is always
> by
> CPU*minute.
>
>> Partition support to reflect network topology
>>
>> CPU hour and job count accounting by group, queue, and user. Some
>> interest in accelerator usage tracking as well.
>
> For accelerator tracking, you'll want to add in the GPU Gres to the relevant
> nodes.
>
> GresTypes=gpu
> AccountingStorageTRES=gres/gpu
>
>> Provide a delay for preempted jobs to checkpoint at the application
>> level rather than just be kill -KILL'ed immediately.
>
> I see you have set KillWait=30 which should provide for this.
>
>> Limit number of jobs per user that can accumulate queue time based
>> priority
>
> There's no native way to do this, but a few alternatives:
>
> - Set the various PriorityWeight values to favor other values (usually
> Fairshare) more than Age.
>
> - Limit the number of jobs that user can submit to that partition using a
> PartitionQOS.
>
> ________________________________
> You are receiving this mail because:
>
> You reported the bug.
Comment 4 Tim Wickberg 2016-08-25 14:47:38 MDT
(In reply to Stephen Fralich from comment #3)
> I have a question about floating paritions. Let say:
> I have 10 nodes with 128GB and 15 nodes with 256GB
> 
> groupA owns 3 128GB nodes and 2 256GB nodes
> groupB owns 7 128GB nodes 13 256GB nodes
> 
> Is there a way for groupA to run a single job using 5 nodes? It seems
> like the nodes with different memory configurations would have to be
> in different partitions.

Partitions can contain whatever mix of nodes you want.

Generally, you'd want users to specify memory requirements with --mem if you do this though.

You can also use the 'Features' settings on the nodes to give users a way to filter between hardware types within a partition with --constraint, e.g. if you have a mix of different CPU types as well.
Comment 5 Stephen Fralich 2016-08-25 15:33:36 MDT
I only want groupA to be able to use a maximum of 3 128GB nodes and
maximum of 2 256GB nodes at a time.

The only way I can see to do this is create a partition with all the
128GB nodes for groupA and another partition with all the 256GB nodes
for groupA then use the GrpNodes parameter to designate the number
they should be able to use.

If there's some better way to approach this, I'd be interested to hear. Thanks,

Stephen

On Thu, Aug 25, 2016 at 1:47 PM,  <bugs@schedmd.com> wrote:
> Comment # 4 on bug 3013 from Tim Wickberg
>
> (In reply to Stephen Fralich from comment #3)
>> I have a question about floating paritions. Let say:
>> I have 10 nodes with 128GB and 15 nodes with 256GB
>>
>> groupA owns 3 128GB nodes and 2 256GB nodes
>> groupB owns 7 128GB nodes 13 256GB nodes
>>
>> Is there a way for groupA to run a single job using 5 nodes? It seems
>> like the nodes with different memory configurations would have to be
>> in different partitions.
>
> Partitions can contain whatever mix of nodes you want.
>
> Generally, you'd want users to specify memory requirements with --mem if you
> do
> this though.
>
> You can also use the 'Features' settings on the nodes to give users a way to
> filter between hardware types within a partition with --constraint, e.g. if
> you
> have a mix of different CPU types as well.
>
> ________________________________
> You are receiving this mail because:
>
> You reported the bug.
Comment 6 Tim Wickberg 2016-08-25 16:31:38 MDT
(In reply to Stephen Fralich from comment #5)
> I only want groupA to be able to use a maximum of 3 128GB nodes and
> maximum of 2 256GB nodes at a time.
> 
> The only way I can see to do this is create a partition with all the
> 128GB nodes for groupA and another partition with all the 256GB nodes
> for groupA then use the GrpNodes parameter to designate the number
> they should be able to use.
> 
> If there's some better way to approach this, I'd be interested to hear.
> Thanks

You're correct in that there's nothing to let you constrain heterogenous node types in a single partition currently. In this case sticking with fixed partition definitions may make sense.

Is there anything else I can answer on this?
Comment 7 Stephen Fralich 2016-08-30 12:02:05 MDT
No, thanks. I'll probably have new different questions, but I can open
a new request for those when they arise. Thanks again.

On Thu, Aug 25, 2016 at 3:31 PM,  <bugs@schedmd.com> wrote:
> Comment # 6 on bug 3013 from Tim Wickberg
>
> (In reply to Stephen Fralich from comment #5)
>> I only want groupA to be able to use a maximum of 3 128GB nodes and
>> maximum of 2 256GB nodes at a time.
>>
>> The only way I can see to do this is create a partition with all the
>> 128GB nodes for groupA and another partition with all the 256GB nodes
>> for groupA then use the GrpNodes parameter to designate the number
>> they should be able to use.
>>
>> If there's some better way to approach this, I'd be interested to hear.
>> Thanks
>
> You're correct in that there's nothing to let you constrain heterogenous
> node
> types in a single partition currently. In this case sticking with fixed
> partition definitions may make sense.
>
> Is there anything else I can answer on this?
>
> ________________________________
> You are receiving this mail because:
>
> You reported the bug.
Comment 8 Tim Wickberg 2016-08-30 12:17:44 MDT
(In reply to Stephen Fralich from comment #7)
> No, thanks. I'll probably have new different questions, but I can open
> a new request for those when they arise. Thanks again.

You're certainly welcome. Marking resolved/infogiven.