| Summary: | SC15 Configuration Suggestions | ||
|---|---|---|---|
| Product: | Slurm | Reporter: | Stephen Fralich <sjf4> |
| Component: | Configuration | Assignee: | Tim Wickberg <tim> |
| Status: | RESOLVED INFOGIVEN | QA Contact: | |
| Severity: | 4 - Minor Issue | ||
| Priority: | --- | ||
| Version: | 16.05.4 | ||
| Hardware: | Linux | ||
| OS: | Linux | ||
| Site: | University of Washington | Slinky Site: | --- |
| Alineos Sites: | --- | Atos/Eviden Sites: | --- |
| Confidential Site: | --- | Coreweave sites: | --- |
| Cray Sites: | --- | DS9 clusters: | --- |
| Google sites: | --- | HPCnow Sites: | --- |
| HPE Sites: | --- | IBM Sites: | --- |
| NOAA SIte: | --- | NoveTech Sites: | --- |
| Nvidia HWinf-CS Sites: | --- | OCF Sites: | --- |
| Recursion Pharma Sites: | --- | SFW Sites: | --- |
| SNIC sites: | --- | Tzag Elita Sites: | --- |
| Linux Distro: | --- | Machine Name: | |
| CLE Version: | Version Fixed: | ||
| Target Release: | --- | DevPrio: | --- |
| Emory-Cloud Sites: | --- | ||
| Attachments: | Current test configuration | ||
Moving to Sev4; please always submit new issues as Sev1-4. Sev5 is longer-term feature requests and won't get updated as promptly. I'm working on a more in-depth response to this. But one thing to point you at is the QOS documentation page which describes the PartitionQOS feature that could be used to implement "floating" partitions, where each has a set resource limit but is allowed access to any of the underlying hardware. http://slurm.schedmd.com/qos.html (In reply to Stephen Fralich from comment #0) > Created attachment 3428 [details] > Current test configuration > > At Supercomputing last year you all suggested we make pools of common > hardware nodes allowing a group to use up to a certain number rather than > assign specific node to specific groups. I'm interested in adopting that > model. > > I've attached the test configuration I've been using previously. It > implements the system I originally planned on before our conversation at > Supercomputing. > > Here's the text of a description of our requirements I've previously sent in > e-email before as well: > We have a condo cluster, so specific groups own specific nodes. > > Node owners jobs run on their own nodes by default. > > Other users can run on other group's nodes via a preemptee queue, but > must be preempted within about a minute if the owner needs the > resource > > A user's priority in the preemptee queue is determined by the number > of nodes his/her group owns and group usage > > Optional usage based priority on some group's nodes > > We have a preference for per node scheduling rather than per core. > Being able to designate some nodes as optionally per-core scheduled if > jobs are coming from a given queue is important. So in other words, > when the node is running a preemptee job, we want it per node > scheduled, but for some nodes, while the owner is using the node we > want it per-core scheduled. > > Our system is heterogeneous: we have nodes with 8, 12, 16, and 32 > cores. It's useful for users to be able to request 1 or more nodes > with any number of cores, receive a set of nodes with the same core > count, and have the accounting reflect they used the entire node. > > Partition support to reflect network topology > > CPU hour and job count accounting by group, queue, and user. Some > interest in accelerator usage tracking as well. > > Provide a delay for preempted jobs to checkpoint at the application > level rather than just be kill -KILL'ed immediately. > > Limit number of jobs per user that can accumulate queue time based priority My responses are inline below interspersed with some configuration values to consider adding to slurm.conf. Please let me know if I can elaborate further on any of these. - Tim (In reply to Stephen Fralich from comment #0) > Created attachment 3428 [details] > Current test configuration > > At Supercomputing last year you all suggested we make pools of common > hardware nodes allowing a group to use up to a certain number rather than > assign specific node to specific groups. I'm interested in adopting that > model. The 'floating' partitions can be implemented with what is called a PartitionQOS. The QOS describes the relevant resource limits for that partition, then Slurm will keep track and only allow that partition to access up to that amount of resources in the nodes that are made available to it. http://slurm.schedmd.com/qos.html > I've attached the test configuration I've been using previously. It > implements the system I originally planned on before our conversation at > Supercomputing. > > Here's the text of a description of our requirements I've previously sent in > e-email before as well: > We have a condo cluster, so specific groups own specific nodes. > > Node owners jobs run on their own nodes by default. > > Other users can run on other group's nodes via a preemptee queue, but > must be preempted within about a minute if the owner needs the > resource http://slurm.schedmd.com/preempt.html It looks like you already have this in place. > A user's priority in the preemptee queue is determined by the number > of nodes his/her group owns and group usage Priority between users is best established through FairShare scheduling. You'd want to structure the fairshare values between the different accounts in sacctmgr appropriately. I would recommend using the Fair_Tree algorithm developed by BYU in place of traditional FairShare. http://slurm.schedmd.com/fair_tree.html (Just set PriorityFlags=FAIR_TREE) > Optional usage based priority on some group's nodes Fairshare will still sort between users in the account on all partitions, although this is obviously only going to impact users in that account. One caveat: usage under a group's own nodes would accumulate and be factored in the global fair share calculation. If you'd prefer that a group's usage on their own nodes be disregarded, you can set TRESBillingWeights=CPU=0,Mem=0,GRES/gpu=0 on those partitions. > We have a preference for per node scheduling rather than per core. > Being able to designate some nodes as optionally per-core scheduled if > jobs are coming from a given queue is important. So in other words, > when the node is running a preemptee job, we want it per node > scheduled, but for some nodes, while the owner is using the node we > want it per-core scheduled. Setting OverSubscribe=Exclusive on the partition would force that partition to only schedule full nodes. It looks like you already have this in effect. ('Shared' was renamed to 'OverSubscribe' in the 16.05 release although both terms still function.) > Our system is heterogeneous: we have nodes with 8, 12, 16, and 32 > cores. It's useful for users to be able to request 1 or more nodes > with any number of cores, receive a set of nodes with the same core > count, and have the accounting reflect they used the entire node. Accounting should reflect the full node worth of CPUs. Accounting is always by CPU*minute. > Partition support to reflect network topology > > CPU hour and job count accounting by group, queue, and user. Some > interest in accelerator usage tracking as well. For accelerator tracking, you'll want to add in the GPU Gres to the relevant nodes. GresTypes=gpu AccountingStorageTRES=gres/gpu > Provide a delay for preempted jobs to checkpoint at the application > level rather than just be kill -KILL'ed immediately. I see you have set KillWait=30 which should provide for this. > Limit number of jobs per user that can accumulate queue time based priority There's no native way to do this, but a few alternatives: - Set the various PriorityWeight values to favor other values (usually Fairshare) more than Age. - Limit the number of jobs that user can submit to that partition using a PartitionQOS. I have a question about floating paritions. Let say: I have 10 nodes with 128GB and 15 nodes with 256GB groupA owns 3 128GB nodes and 2 256GB nodes groupB owns 7 128GB nodes 13 256GB nodes Is there a way for groupA to run a single job using 5 nodes? It seems like the nodes with different memory configurations would have to be in different partitions. Let me know if I'm misunderstanding. Thanks, Stephen On Tue, Aug 23, 2016 at 8:36 AM, <bugs@schedmd.com> wrote: > Comment # 2 on bug 3013 from Tim Wickberg > > My responses are inline below interspersed with some configuration values to > consider adding to slurm.conf. > > Please let me know if I can elaborate further on any of these. > > - Tim > > (In reply to Stephen Fralich from comment #0) >> Created attachment 3428 [details] >> Current test configuration >> >> At Supercomputing last year you all suggested we make pools of common >> hardware nodes allowing a group to use up to a certain number rather than >> assign specific node to specific groups. I'm interested in adopting that >> model. > > The 'floating' partitions can be implemented with what is called a > PartitionQOS. > > The QOS describes the relevant resource limits for that partition, then > Slurm > will keep track and only allow that partition to access up to that amount of > resources in the nodes that are made available to it. > > http://slurm.schedmd.com/qos.html > >> I've attached the test configuration I've been using previously. It >> implements the system I originally planned on before our conversation at >> Supercomputing. >> >> Here's the text of a description of our requirements I've previously sent >> in >> e-email before as well: >> We have a condo cluster, so specific groups own specific nodes. >> >> Node owners jobs run on their own nodes by default. >> >> Other users can run on other group's nodes via a preemptee queue, but >> must be preempted within about a minute if the owner needs the >> resource > > http://slurm.schedmd.com/preempt.html > > It looks like you already have this in place. > >> A user's priority in the preemptee queue is determined by the number >> of nodes his/her group owns and group usage > > Priority between users is best established through FairShare scheduling. > You'd want to structure the fairshare values between the different accounts > in sacctmgr appropriately. > > I would recommend using the Fair_Tree algorithm developed by BYU in place of > traditional FairShare. http://slurm.schedmd.com/fair_tree.html > > (Just set PriorityFlags=FAIR_TREE) > >> Optional usage based priority on some group's nodes > > Fairshare will still sort between users in the account on all partitions, > although this is obviously only going to impact users in that account. > > One caveat: usage under a group's own nodes would accumulate and be factored > in the global fair share calculation. If you'd prefer that a group's usage > on > their own nodes be disregarded, you can set > > TRESBillingWeights=CPU=0,Mem=0,GRES/gpu=0 on those partitions. > >> We have a preference for per node scheduling rather than per core. >> Being able to designate some nodes as optionally per-core scheduled if >> jobs are coming from a given queue is important. So in other words, >> when the node is running a preemptee job, we want it per node >> scheduled, but for some nodes, while the owner is using the node we >> want it per-core scheduled. > > Setting OverSubscribe=Exclusive on the partition would force that partition > to > only schedule full nodes. It looks like you already have this in effect. > ('Shared' was renamed to 'OverSubscribe' in the 16.05 release although both > terms still function.) > >> Our system is heterogeneous: we have nodes with 8, 12, 16, and 32 >> cores. It's useful for users to be able to request 1 or more nodes >> with any number of cores, receive a set of nodes with the same core >> count, and have the accounting reflect they used the entire node. > > Accounting should reflect the full node worth of CPUs. Accounting is always > by > CPU*minute. > >> Partition support to reflect network topology >> >> CPU hour and job count accounting by group, queue, and user. Some >> interest in accelerator usage tracking as well. > > For accelerator tracking, you'll want to add in the GPU Gres to the relevant > nodes. > > GresTypes=gpu > AccountingStorageTRES=gres/gpu > >> Provide a delay for preempted jobs to checkpoint at the application >> level rather than just be kill -KILL'ed immediately. > > I see you have set KillWait=30 which should provide for this. > >> Limit number of jobs per user that can accumulate queue time based >> priority > > There's no native way to do this, but a few alternatives: > > - Set the various PriorityWeight values to favor other values (usually > Fairshare) more than Age. > > - Limit the number of jobs that user can submit to that partition using a > PartitionQOS. > > ________________________________ > You are receiving this mail because: > > You reported the bug. (In reply to Stephen Fralich from comment #3) > I have a question about floating paritions. Let say: > I have 10 nodes with 128GB and 15 nodes with 256GB > > groupA owns 3 128GB nodes and 2 256GB nodes > groupB owns 7 128GB nodes 13 256GB nodes > > Is there a way for groupA to run a single job using 5 nodes? It seems > like the nodes with different memory configurations would have to be > in different partitions. Partitions can contain whatever mix of nodes you want. Generally, you'd want users to specify memory requirements with --mem if you do this though. You can also use the 'Features' settings on the nodes to give users a way to filter between hardware types within a partition with --constraint, e.g. if you have a mix of different CPU types as well. I only want groupA to be able to use a maximum of 3 128GB nodes and maximum of 2 256GB nodes at a time. The only way I can see to do this is create a partition with all the 128GB nodes for groupA and another partition with all the 256GB nodes for groupA then use the GrpNodes parameter to designate the number they should be able to use. If there's some better way to approach this, I'd be interested to hear. Thanks, Stephen On Thu, Aug 25, 2016 at 1:47 PM, <bugs@schedmd.com> wrote: > Comment # 4 on bug 3013 from Tim Wickberg > > (In reply to Stephen Fralich from comment #3) >> I have a question about floating paritions. Let say: >> I have 10 nodes with 128GB and 15 nodes with 256GB >> >> groupA owns 3 128GB nodes and 2 256GB nodes >> groupB owns 7 128GB nodes 13 256GB nodes >> >> Is there a way for groupA to run a single job using 5 nodes? It seems >> like the nodes with different memory configurations would have to be >> in different partitions. > > Partitions can contain whatever mix of nodes you want. > > Generally, you'd want users to specify memory requirements with --mem if you > do > this though. > > You can also use the 'Features' settings on the nodes to give users a way to > filter between hardware types within a partition with --constraint, e.g. if > you > have a mix of different CPU types as well. > > ________________________________ > You are receiving this mail because: > > You reported the bug. (In reply to Stephen Fralich from comment #5) > I only want groupA to be able to use a maximum of 3 128GB nodes and > maximum of 2 256GB nodes at a time. > > The only way I can see to do this is create a partition with all the > 128GB nodes for groupA and another partition with all the 256GB nodes > for groupA then use the GrpNodes parameter to designate the number > they should be able to use. > > If there's some better way to approach this, I'd be interested to hear. > Thanks You're correct in that there's nothing to let you constrain heterogenous node types in a single partition currently. In this case sticking with fixed partition definitions may make sense. Is there anything else I can answer on this? No, thanks. I'll probably have new different questions, but I can open a new request for those when they arise. Thanks again. On Thu, Aug 25, 2016 at 3:31 PM, <bugs@schedmd.com> wrote: > Comment # 6 on bug 3013 from Tim Wickberg > > (In reply to Stephen Fralich from comment #5) >> I only want groupA to be able to use a maximum of 3 128GB nodes and >> maximum of 2 256GB nodes at a time. >> >> The only way I can see to do this is create a partition with all the >> 128GB nodes for groupA and another partition with all the 256GB nodes >> for groupA then use the GrpNodes parameter to designate the number >> they should be able to use. >> >> If there's some better way to approach this, I'd be interested to hear. >> Thanks > > You're correct in that there's nothing to let you constrain heterogenous > node > types in a single partition currently. In this case sticking with fixed > partition definitions may make sense. > > Is there anything else I can answer on this? > > ________________________________ > You are receiving this mail because: > > You reported the bug. (In reply to Stephen Fralich from comment #7) > No, thanks. I'll probably have new different questions, but I can open > a new request for those when they arise. Thanks again. You're certainly welcome. Marking resolved/infogiven. |
Created attachment 3428 [details] Current test configuration At Supercomputing last year you all suggested we make pools of common hardware nodes allowing a group to use up to a certain number rather than assign specific node to specific groups. I'm interested in adopting that model. I've attached the test configuration I've been using previously. It implements the system I originally planned on before our conversation at Supercomputing. Here's the text of a description of our requirements I've previously sent in e-email before as well: We have a condo cluster, so specific groups own specific nodes. Node owners jobs run on their own nodes by default. Other users can run on other group's nodes via a preemptee queue, but must be preempted within about a minute if the owner needs the resource A user's priority in the preemptee queue is determined by the number of nodes his/her group owns and group usage Optional usage based priority on some group's nodes We have a preference for per node scheduling rather than per core. Being able to designate some nodes as optionally per-core scheduled if jobs are coming from a given queue is important. So in other words, when the node is running a preemptee job, we want it per node scheduled, but for some nodes, while the owner is using the node we want it per-core scheduled. Our system is heterogeneous: we have nodes with 8, 12, 16, and 32 cores. It's useful for users to be able to request 1 or more nodes with any number of cores, receive a set of nodes with the same core count, and have the accounting reflect they used the entire node. Partition support to reflect network topology CPU hour and job count accounting by group, queue, and user. Some interest in accelerator usage tracking as well. Provide a delay for preempted jobs to checkpoint at the application level rather than just be kill -KILL'ed immediately. Limit number of jobs per user that can accumulate queue time based priority