Ticket 6548

Hi Naveed,

I think you may be able to accomplish what you want to do with overlapping reservations.  You can create a reservation that covers the whole node for your general cpu jobs and create an overlapping reservation to reserve a few cpus to go with the gpus you have.  I set up example reservations on a test node, in my example I just made them be for different users:

$ scontrol create reservation starttime=now duration=2:00:00 nodes=node01 user=user1 tres=cpu=4
Reservation created: user1_28

$ scontrol create reservation starttime=now duration=2:00:00 nodes=node01 user=ben tres=cpu=2
Reservation created: ben_29

$ scontrol show reservation
ReservationName=user1_28 StartTime=2019-02-20T16:45:06 EndTime=2019-02-20T18:45:06 Duration=02:00:00
   Nodes=node01 NodeCnt=1 CoreCnt=4 Features=(null) PartitionName=debug Flags=SPEC_NODES
     NodeName=node01 CoreIDs=0-3
   TRES=cpu=8
   Users=user1 Accounts=(null) Licenses=(null) State=ACTIVE BurstBuffer=(null) Watts=n/a

ReservationName=ben_29 StartTime=2019-02-20T16:45:15 EndTime=2019-02-20T18:45:15 Duration=02:00:00
   Nodes=node01 NodeCnt=1 CoreCnt=2 Features=(null) PartitionName=debug Flags=SPEC_NODES
     NodeName=node01 CoreIDs=4-5
   TRES=cpu=4
   Users=ben Accounts=(null) Licenses=(null) State=ACTIVE BurstBuffer=(null) Watts=n/a




With those two reservations in place I submitted jobs to the first reservation as 'user1' to verify that it would use 8 cpus but not more:

user1@ben-XPS-15-9570:~$ sbatch -n4 -wnode01 --reservation=user1_28 --wrap="sleep 300"
Submitted batch job 2178
user1@ben-XPS-15-9570:~$ sbatch -n4 -wnode01 --reservation=user1_28 --wrap="sleep 300"
Submitted batch job 2179
user1@ben-XPS-15-9570:~$ sbatch -n4 -wnode01 --reservation=user1_28 --wrap="sleep 300"
Submitted batch job 2180

user1@ben-XPS-15-9570:~$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              2180     debug     wrap    user1 PD       0:00      1 (Resources)
              2179     debug     wrap    user1  R       0:01      1 node01
              2178     debug     wrap    user1  R       0:04      1 node01



I then submitted jobs to the other reservation as 'ben' and requested gpus and verified that they would occupy the other cpus on the node:

ben@ben-XPS-15-9570:~/slurm$ sbatch -n2 -wnode01 --reservation=ben_29 --gres=gpu:2 --wrap="sleep 300"
Submitted batch job 2181
ben@ben-XPS-15-9570:~/slurm$ sbatch -n2 -wnode01 --reservation=ben_29 --gres=gpu:2 --wrap="sleep 300"
Submitted batch job 2182
ben@ben-XPS-15-9570:~/slurm$ sbatch -n2 -wnode01 --reservation=ben_29 --gres=gpu:2 --wrap="sleep 300"
Submitted batch job 2183

ben@ben-XPS-15-9570:~/slurm$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              2183     debug     wrap      ben PD       0:00      1 (Resources)
              2180     debug     wrap    user1 PD       0:00      1 (ReqNodeNotAvail, UnavailableNodes:node01)
              2182     debug     wrap      ben  R       0:04      1 node01
              2181     debug     wrap      ben  R       0:06      1 node01
              2179     debug     wrap    user1  R       0:57      1 node01
              2178     debug     wrap    user1  R       1:00      1 node01




You mention that you have all your nodes in a single partition.  Have you thought of creating a partition for your gpu nodes?  You can leave the nodes in the primary partition as well, but with different partitions it would be easier to separate the reservations.  

Let me know if you have questions about this.

Thanks,
Ben

Thank you.  That wasn’t exactly what I was looking for.  Overlapping reservations would make it more complicated for my users, ad I am trying to make it simpler. I was hoping there would be a way to tie a particular core to a particular gpu so that one couldn’t be requested without the other to ensure that a core is always available for a request of a gpu while still allowing use of the rest of the node since we have very heavy cpu users.

Any thoughts on other ways to accomplish this?  Do you know of other sites that do similar?

Can a part of a  node be in one partition and the rest in another? I assume not, but perhaps there is some other wy to accomplish it.

Naveed

From: "bugs@schedmd.com" <bugs@schedmd.com>
Date: Wednesday, February 20, 2019 at 3:08 PM
To: Naveed Near-Ansari <naveed@caltech.edu>
Subject: [Bug 6548] tying gpu allocation to reserved cpus

Comment # 2<https://bugs.schedmd.com/show_bug.cgi?id=6548#c2> on bug 6548<https://bugs.schedmd.com/show_bug.cgi?id=6548> from Ben<mailto:ben@schedmd.com>

Hi Naveed,



I think you may be able to accomplish what you want to do with overlapping

reservations.  You can create a reservation that covers the whole node for your

general cpu jobs and create an overlapping reservation to reserve a few cpus to

go with the gpus you have.  I set up example reservations on a test node, in my

example I just made them be for different users:



$ scontrol create reservation starttime=now duration=2:00:00 nodes=node01

user=user1 tres=cpu=4

Reservation created: user1_28



$ scontrol create reservation starttime=now duration=2:00:00 nodes=node01

user=ben tres=cpu=2

Reservation created: ben_29



$ scontrol show reservation

ReservationName=user1_28 StartTime=2019-02-20T16:45:06

EndTime=2019-02-20T18:45:06 Duration=02:00:00

   Nodes=node01 NodeCnt=1 CoreCnt=4 Features=(null) PartitionName=debug

Flags=SPEC_NODES

     NodeName=node01 CoreIDs=0-3

   TRES=cpu=8

   Users=user1 Accounts=(null) Licenses=(null) State=ACTIVE BurstBuffer=(null)

Watts=n/a



ReservationName=ben_29 StartTime=2019-02-20T16:45:15

EndTime=2019-02-20T18:45:15 Duration=02:00:00

   Nodes=node01 NodeCnt=1 CoreCnt=2 Features=(null) PartitionName=debug

Flags=SPEC_NODES

     NodeName=node01 CoreIDs=4-5

   TRES=cpu=4

   Users=ben Accounts=(null) Licenses=(null) State=ACTIVE BurstBuffer=(null)

Watts=n/a









With those two reservations in place I submitted jobs to the first reservation

as 'user1' to verify that it would use 8 cpus but not more:



user1@ben-XPS-15-9570:~$ sbatch -n4 -wnode01 --reservation=user1_28

--wrap="sleep 300"

Submitted batch job 2178

user1@ben-XPS-15-9570:~$ sbatch -n4 -wnode01 --reservation=user1_28

--wrap="sleep 300"

Submitted batch job 2179

user1@ben-XPS-15-9570:~$ sbatch -n4 -wnode01 --reservation=user1_28

--wrap="sleep 300"

Submitted batch job 2180



user1@ben-XPS-15-9570:~$ squeue

             JOBID PARTITION     NAME     USER ST       TIME  NODES

NODELIST(REASON)

              2180     debug     wrap    user1 PD       0:00      1 (Resources)

              2179     debug     wrap    user1  R       0:01      1 node01

              2178     debug     wrap    user1  R       0:04      1 node01







I then submitted jobs to the other reservation as 'ben' and requested gpus and

verified that they would occupy the other cpus on the node:



ben@ben-XPS-15-9570:~/slurm$ sbatch -n2 -wnode01 --reservation=ben_29

--gres=gpu:2 --wrap="sleep 300"

Submitted batch job 2181

ben@ben-XPS-15-9570:~/slurm$ sbatch -n2 -wnode01 --reservation=ben_29

--gres=gpu:2 --wrap="sleep 300"

Submitted batch job 2182

ben@ben-XPS-15-9570:~/slurm$ sbatch -n2 -wnode01 --reservation=ben_29

--gres=gpu:2 --wrap="sleep 300"

Submitted batch job 2183



ben@ben-XPS-15-9570:~/slurm$ squeue

             JOBID PARTITION     NAME     USER ST       TIME  NODES

NODELIST(REASON)

              2183     debug     wrap      ben PD       0:00      1 (Resources)

              2180     debug     wrap    user1 PD       0:00      1

(ReqNodeNotAvail, UnavailableNodes:node01)

              2182     debug     wrap      ben  R       0:04      1 node01

              2181     debug     wrap      ben  R       0:06      1 node01

              2179     debug     wrap    user1  R       0:57      1 node01

              2178     debug     wrap    user1  R       1:00      1 node01









You mention that you have all your nodes in a single partition.  Have you

thought of creating a partition for your gpu nodes?  You can leave the nodes in

the primary partition as well, but with different partitions it would be easier

to separate the reservations.



Let me know if you have questions about this.



Thanks,

Ben

________________________________
You are receiving this mail because:

  *   You reported the bug.

Hi Naveed,

Yes, I was trying to come up with a way to do this in a single partition initially, but it sounds like you are open to creating a separate partition for the gpu jobs.  If this is the case you can use the MaxCPUsPerNode parameter to define how many cpus each partition can use.  Here's the description of this parameter from the documentation:

MaxCPUsPerNode
    Maximum number of CPUs on any node available to all jobs from this partition. This can be especially useful to schedule GPUs. For example a node can be associated with two Slurm partitions (e.g. "cpu" and "gpu") and the partition/queue "cpu" could be limited to only a subset of the node's CPUs, ensuring that one or more CPUs would be available to jobs in the "gpu" partition/queue. 


You would define this in the slurm.conf on the partition line, so it looked something like this:
PartitionName=cpu Nodes=node[01-10] Default=YES MaxTime=INFINITE MaxCPUsPerNode=20 State=UP
PartitionName=gpu Nodes=node[01-10] Default=NO MaxTime=INFINITE MaxCPUsPerNode=4 State=UP

You can find the information about this parameter from the documentation here:
https://slurm.schedmd.com/slurm.conf.html

Along with this you can associate GPUs with certain CPUs in your gres.conf file.  It could look something like this:
Name=gpu Type=gtx560 File=/dev/nvidia0 COREs=0,1
Name=gpu Type=gtx560 File=/dev/nvidia1 COREs=0,1
Name=gpu Type=tesla File=/dev/nvidia2 COREs=2,3
Name=gpu Type=tesla File=/dev/nvidia3 COREs=2,3 

This example comes from the gres.conf documentation:
https://slurm.schedmd.com/gres.conf.html


I hope this is closer to what you are looking for.  Let me know if you have questions about it.

Thanks,
Ben

That looks closer, but not quite there.  Can you set MaxCPUsPerNode per node in the partition?  I ask since there are both cpu and gpu nodes that would be in the “any” queue, each of which  have different core counts.

Something like this perhaps?

PartitionName=any Nodes= hpc-22-[07-24],hpc-23-[07-24],hpc-24-[07-24],hpc-25-[03-10],hpc-89-[03-26],hpc-90-[03-26,29-30],hpc-91-[09-21,24-25],hpc-92-[03-26,29-30],hpc-93-[03-26,29-30] MaxCPUsPerNode=32 Nodes= hpc-22-[28,30,32,34,36,38],hpc-23-[28,30,32,34,36,38],hpc-24-[28,30,32,34,36,38],hpc-25-[14-15,17-18,20-21,23-24],hpc-26-[14-15,17-18,20-21,23-24],hpc-89-[35-38],hpc-90-[35-38],hpc-91-[32-33],hpc-92-[36-38],hpc-93-[36-38] MaxCPUsPerNode=20 MaxTime=14-0 State=UP


Or possibly a QOS tied to a partition that has maximums?

Not sure the best way or the other choices.  This is defeintely a step in the right direction as far as simplicity to the end user goes.

Naveed

From: "bugs@schedmd.com" <bugs@schedmd.com>
Date: Tuesday, February 26, 2019 at 8:28 AM
To: Naveed Near-Ansari <naveed@caltech.edu>
Subject: [Bug 6548] tying gpu allocation to reserved cpus

Comment # 5<https://bugs.schedmd.com/show_bug.cgi?id=6548#c5> on bug 6548<https://bugs.schedmd.com/show_bug.cgi?id=6548> from Ben<mailto:ben@schedmd.com>

Hi Naveed,



Yes, I was trying to come up with a way to do this in a single partition

initially, but it sounds like you are open to creating a separate partition for

the gpu jobs.  If this is the case you can use the MaxCPUsPerNode parameter to

define how many cpus each partition can use.  Here's the description of this

parameter from the documentation:



MaxCPUsPerNode

    Maximum number of CPUs on any node available to all jobs from this

partition. This can be especially useful to schedule GPUs. For example a node

can be associated with two Slurm partitions (e.g. "cpu" and "gpu") and the

partition/queue "cpu" could be limited to only a subset of the node's CPUs,

ensuring that one or more CPUs would be available to jobs in the "gpu"

partition/queue.





You would define this in the slurm.conf on the partition line, so it looked

something like this:

PartitionName=cpu Nodes=node[01-10] Default=YES MaxTime=INFINITE

MaxCPUsPerNode=20 State=UP

PartitionName=gpu Nodes=node[01-10] Default=NO MaxTime=INFINITE

MaxCPUsPerNode=4 State=UP



You can find the information about this parameter from the documentation here:

https://slurm.schedmd.com/slurm.conf.html



Along with this you can associate GPUs with certain CPUs in your gres.conf

file.  It could look something like this:

Name=gpu Type=gtx560 File=/dev/nvidia0 COREs=0,1

Name=gpu Type=gtx560 File=/dev/nvidia1 COREs=0,1

Name=gpu Type=tesla File=/dev/nvidia2 COREs=2,3

Name=gpu Type=tesla File=/dev/nvidia3 COREs=2,3



This example comes from the gres.conf documentation:

https://slurm.schedmd.com/gres.conf.html





I hope this is closer to what you are looking for.  Let me know if you have

questions about it.



Thanks,

Ben

________________________________
You are receiving this mail because:

  *   You reported the bug.

Hi Naveed,

Unfortunately you can't have multiple 'Nodes=' definitions in a single partition.  Slurm will complain about the syntax and just use the latest value in the config line, ignoring the first.  

I was looking at whether you could configure something at the qos level to enforce this, but there isn't an equivalent setting for a qos.  The closest is MaxTRESPerNode, but this enforces the maximum number of resources a single job can request per node, but if individual jobs are below this limit and there's room for multiple jobs in the same partition on a node it will consume all the available resources on the node.  

What you can do is define unique partitions for the cpu and gpu nodes so you can define the MaxCPUsPerNode appropriate for each type of node.  Then the users can request that their jobs run in either of the partitions when they submit, like this:
sbatch -p debug,gpu test.job

This will allow the job to go to whichever partition is available to run the job first.  This does require the user to specify both partitions at submission time, but that should be easier to do than requesting a reservation at submit time.  

Let me know if this looks like it will work.

Thanks,
Ben

Hi Naveed,

I wanted to follow up and see if you had additional questions about the information I sent.  Does using multiple partitions work for you?  Let me know if you still need help with this ticket.

Thanks,
Ben

Hi Naveed,

I have a colleague with a similar request and we were discussing how to best address a situation where you want to have certain CPUs/cores reserved for your gpus.  The idea we decided would probably work best is to create another gres that you can associate with the CPUs not associated with your GPU gres.  Then you can use a submit plugin to assign the "nongpu" gres unless they request the "gpu" gres.

Here is an example configuration I used for testing.  In my slurm.conf these are the relevant lines that need to be there (I used node10 for this test):
...
TaskPlugin=task/affinity,task/cgroup
GresTypes=gpu,test
NodeName=node10  Gres=gpu:1,test:1 CPUs=12 SocketsPerBoard=1 CoresPerSocket=6 ThreadsPerCore=2
...


In my gres.conf I needed to specify which cores would be bound to each gres:
NodeName=node10 Name=gpu Count=1 File=/dev/nvidia0 Cores=0
NodeName=node10 Name=test Count=1 File=/dev/zero Cores=2


Then when I run a job that requests one of those gres' I can see that I'm limited to the CPUs associated with the cores I tied the gres to:
$ srun -n1 --gres=test:1 cat /proc/self/status | grep Cpus_allowed
Cpus_allowed:   082
Cpus_allowed_list:      1,7

$ srun -n1 --gres=gpu:1 cat /proc/self/status | grep Cpus_allowed
Cpus_allowed:   041
Cpus_allowed_list:      0,6


One thing to note is that I had to have a 'File=' defined in my gres.conf for this to work properly.  If I just specified a count of the number of generic resources I wanted it would go anywhere on the cluster.  

There is also a feature request that has been opened to allow you to accomplish the same thing without having to create a second gres.  You can follow the status of this request here:
https://bugs.schedmd.com/show_bug.cgi?id=6734

Let me know if you have questions about implementing a secondary gres.

Thanks,
Ben

Hi Naveed,

I wanted to follow up and make sure you were able to use the suggestion I sent of using the gres.conf file to allow you to tie the gres's to certain cores on the system.  Let me know if you still need help with this ticket or if it's ok to close.

Thanks,
Ben

Hi Naveed,

I haven't heard from you in a while on this ticket and I think the information I sent should have addressed your question.  I'm going to close this ticket but feel free to let me know if you have additional questions about it.  

Thanks,
Ben

This seems like a potential solution in the short term.  Do you have a example of the submit script you used on it?

In this example, the node has 12 cores, correct?  I see it has a gres for the gpus and a gres for the cpus, but how are they associated? Id this done in the script?

Thanks

Naveed

PS – sorry I took so long to get back in touch with you.

From: "bugs@schedmd.com" <bugs@schedmd.com>
Date: Thursday, March 21, 2019 at 2:30 PM
To: Naveed Near-Ansari <naveed@caltech.edu>
Subject: [Bug 6548] tying gpu allocation to reserved cpus

Ben<mailto:ben@schedmd.com> changed bug 6548<https://bugs.schedmd.com/show_bug.cgi?id=6548>
What

Removed

Added

See Also



https://bugs.schedmd.com/show_bug.cgi?id=6734

Comment # 9<https://bugs.schedmd.com/show_bug.cgi?id=6548#c9> on bug 6548<https://bugs.schedmd.com/show_bug.cgi?id=6548> from Ben<mailto:ben@schedmd.com>

Hi Naveed,



I have a colleague with a similar request and we were discussing how to best

address a situation where you want to have certain CPUs/cores reserved for your

gpus.  The idea we decided would probably work best is to create another gres

that you can associate with the CPUs not associated with your GPU gres.  Then

you can use a submit plugin to assign the "nongpu" gres unless they request the

"gpu" gres.



Here is an example configuration I used for testing.  In my slurm.conf these

are the relevant lines that need to be there (I used node10 for this test):

...

TaskPlugin=task/affinity,task/cgroup

GresTypes=gpu,test

NodeName=node10  Gres=gpu:1,test:1 CPUs=12 SocketsPerBoard=1 CoresPerSocket=6

ThreadsPerCore=2

...





In my gres.conf I needed to specify which cores would be bound to each gres:

NodeName=node10 Name=gpu Count=1 File=/dev/nvidia0 Cores=0

NodeName=node10 Name=test Count=1 File=/dev/zero Cores=2





Then when I run a job that requests one of those gres' I can see that I'm

limited to the CPUs associated with the cores I tied the gres to:

$ srun -n1 --gres=test:1 cat /proc/self/status | grep Cpus_allowed

Cpus_allowed:   082

Cpus_allowed_list:      1,7



$ srun -n1 --gres=gpu:1 cat /proc/self/status | grep Cpus_allowed

Cpus_allowed:   041

Cpus_allowed_list:      0,6





One thing to note is that I had to have a 'File=' defined in my gres.conf for

this to work properly.  If I just specified a count of the number of generic

resources I wanted it would go anywhere on the cluster.



There is also a feature request that has been opened to allow you to accomplish

the same thing without having to create a second gres.  You can follow the

status of this request here:

https://bugs.schedmd.com/show_bug.cgi?id=6734<show_bug.cgi?id=6734>



Let me know if you have questions about implementing a secondary gres.



Thanks,

Ben

________________________________
You are receiving this mail because:

  *   You reported the bug.

Hi Naveed,

I didn't use a submit script, but here are the steps I took to accomplish this.  These are the relevant lines from my slurm.conf:

...
TaskPlugin=task/affinity,task/cgroup
GresTypes=gpu,test
NodeName=node10  Gres=gpu:1,test:1 CPUs=12 SocketsPerBoard=1 CoresPerSocket=6 ThreadsPerCore=2
...



In my gres.conf I needed to specify which cores would be bound to each gres:

NodeName=node10 Name=gpu Count=1 File=/dev/nvidia0 Cores=0
NodeName=node10 Name=test Count=1 File=/dev/zero Cores=2



Then when I run a job that requests one of those gres' I can see that I'm limited to the CPUs associated with the cores I tied the gres to:

$ srun -n1 --gres=test:1 cat /proc/self/status | grep Cpus_allowed
Cpus_allowed:   082
Cpus_allowed_list:      1,7

$ srun -n1 --gres=gpu:1 cat /proc/self/status | grep Cpus_allowed
Cpus_allowed:   041
Cpus_allowed_list:      0,6



The association of the gres with the CPUs is done in the gres.conf.  You specify the Cores you would like to associate the gres with.  You can use lstopo to see the layout of the processors on your machine.  There is more information about this in the gres.conf documentation:
https://slurm.schedmd.com/gres.conf.html#OPT_Cores

Let me know if you have additional questions about this.

Thanks,
Ben