Hi there, We have a QOS with a MinTRES set to gres/gpu=1: # sacctmgr list qos gpu format=name,mintres%24 Name MinTRES ---------- ------------------------ gpu cpu=1,gres/gpu=1 And we use GRES "types" for GPUs, in gres.conf: NodeName=gpu-13-[1-2] Name=gpu Type=gtx File=/dev/nvidia[0-3] CPUs=0-7 [...] NodeName=gpu-9-[1-2] Name=gpu Type=tesla File=/dev/nvidia[0-3] CPUs=0-7 and in nodes definitions: # scontrol show node gpu-13-1 | grep -i gres Gres=gpu:gtx:8 # scontrol show node gpu-9-1 | grep -i gres Gres=gpu:tesla:8 Turns out that the QOS MinTRES limit rejects jobs that are requesting a specific GPU type, ie: * "srun --gres gpu:1" works * "srun --gres gpu:gtx:1" doesn't work, and results in "srun: error: Unable to allocate resources: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)" Is it expected? Our intention is to have a MinTRES that would force users to request at least 1 GPU, but we don't really care about which type. Thanks! -- Kilian
Kilian, what does your AccountingStorageTRES line look like in your slurm.conf? Since TRES are fine grained (meaning the gtx and tesla would result in 2 different TRES) I am not sure it will work like you expect when specifying the type. I can see what you are after though. What I believe is happening right now is the gres/gpu is what you are tracking, but when the type gets thrown into the mix Slurm is treating it as a completely different TRES. This fine grained implementation probably won't work for you. I'm not sure how to change it and still support the situation where people do what the fine grained TRES, perhaps we could track both gres/gpu as well as gres/gpu:gtx|tesla, and when it comes in we match both gres/gpu as well as the type. That would give us both. Any easier solution might be putting this restriction in a job_submit plugin though. So if they don't request a gpu you just deny the job right there. Would this work?
Hi Danny, (In reply to Danny Auble from comment #1) > Kilian, what does your AccountingStorageTRES line look like in your > slurm.conf? # scontrol show config | grep AccountingStorageTRES AccountingStorageTRES = gres/gpu,cpu,mem,energy,node > Since TRES are fine grained (meaning the gtx and tesla would result in 2 > different TRES) I am not sure it will work like you expect when specifying > the type. I can see what you are after though. I see, that makes sense. > What I believe is happening right now is the gres/gpu is what you are > tracking, but when the type gets thrown into the mix Slurm is treating it as > a completely different TRES. This fine grained implementation probably > won't work for you. I'm not sure how to change it and still support the > situation where people do what the fine grained TRES, perhaps we could track > both gres/gpu as well as gres/gpu:gtx|tesla, and when it comes in we match > both gres/gpu as well as the type. That would give us both. That sounds good, yes. That would allow to match both the generic "gpu" resource as well as the type. I'm not sure you want to go down the route of implementing a hierarchical TRES system to handle TRES "types", similar to GRES types. > Any easier solution might be putting this restriction in a job_submit plugin > though. So if they don't request a gpu you just deny the job right there. > > Would this work? I guess, yes. My current workaround is to make them request a generic Gres, without type (--gres=gpu:1) and add a constraint (-C) to the submission: we define the GPU type in a node feature too, so that gives the same behavior. Maybe I'll drop the Gres types altogether since we have the features anyway. Thanks for the explanation! Cheers, -- Kilian
No problem Kilian, I opened an enhancement on the matter, bug 2482. Let me know if you need any more info on this.
(In reply to Danny Auble from comment #3) > No problem Kilian, I opened an enhancement on the matter, bug 2482. Let me > know if you need any more info on this. Sounds great, thanks Danny! Cheers, -- Kilian
Kilian, I think commits b294f81b59f and 35f7a2628307 will do exactly what you are looking for. Meaning a limit on gres/gpu will work for any gres/gpu:*. It turns out we aren't tracking gres/gpu:* TRES they all get summed up in gres/gpu. This changes in 16.05 as we talked suggested in bug 2482. Let me know if you try it out and how it goes.
