SchedMD - Slurm Support – Full Text Ticket Listing
- Home
- | New
- | Browse
- | Search
- | [?]
- | Reports
- | Help
- | New Account
- | Log In
- | Forgot Password
| Summary: | Packing nodes | ||
|---|---|---|---|
| Product: | Slurm | Reporter: | Tom Wurgler <twurgl> |
| Component: | Configuration | Assignee: | Dominik Bartkiewicz <bart> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | 3 - Medium Impact | ||
| Priority: | --- | CC: | mcmullan, thu-ha.tran |
| Version: | 20.02.4 | ||
| Hardware: | Linux | ||
| OS: | Linux | ||
| See Also: | https://bugs.schedmd.com/show_bug.cgi?id=12605 | ||
| Site: | Goodyear | Alineos Sites: | --- |
| Atos/Eviden Sites: | --- | Confidential Site: | --- |
| Coreweave sites: | --- | Cray Sites: | --- |
| DS9 clusters: | --- | HPCnow Sites: | --- |
| HPE Sites: | --- | IBM Sites: | --- |
| NOAA SIte: | --- | OCF Sites: | --- |
| Recursion Pharma Sites: | --- | SFW Sites: | --- |
| SNIC sites: | --- | Linux Distro: | --- |
| Machine Name: | CLE Version: | ||
| Version Fixed: | 21.08.0pre1 | Target Release: | --- |
| DevPrio: | --- | Emory-Cloud Sites: | --- |
| Attachments: | log after setting the variables | ||
|
Description
Tom Wurgler
2020-12-08 09:18:32 MST
Hey Tom, Can you provide an example of a job submission that doesn't behave as expected? I'm interested in what options are getting passed in so I can properly replicate the situation! I did notice "CR_Pack_Nodes" which should (and so far in my testing seems to work) compact tasks like that if the tasks being launched don't consume the whole allocation. Thanks! --Tim We run a FEA code (that I can’t share). But I think if you use our slurm-common.conf with a slurm.conf that has the partitions etc in it and includes the common file, Then submit a job requiring more than one node worth of cores, you should see the problem. It just did the same kind of thing to another user here on another code. He asked for 960 cores and we have 24 cores per node. So ideally, he should have gotten 40 nodes. He really got 41, so it used most of the cores and left some unused and just added another node… The ideal situation would be that a job would use whole nodes if available, and only split a job up when it has to (ie, not pend). We can arrange a call if I am not very clear. Thanks tom From: bugs@schedmd.com <bugs@schedmd.com> Sent: Wednesday, December 9, 2020 11:50 AM To: Tom Wurgler <twurgl@goodyear.com> Subject: [EXT] [Bug 10391] Packing nodes External Email....WARNING....Think before you click or respond....WARNING Comment # 1<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.schedmd.com%2Fshow_bug.cgi%3Fid%3D10391%23c1&data=04%7C01%7Ctwurgl%40goodyear.com%7Cf9ea50e3731845254dff08d89c628424%7C939e896692854a9a9f040887efe8aae0%7C0%7C0%7C637431294221313118%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=MmY3%2BJascwfJanoBAMv%2BSD2kotwrqiyc0CZYN9Xy3Jg%3D&reserved=0> on bug 10391<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.schedmd.com%2Fshow_bug.cgi%3Fid%3D10391&data=04%7C01%7Ctwurgl%40goodyear.com%7Cf9ea50e3731845254dff08d89c628424%7C939e896692854a9a9f040887efe8aae0%7C0%7C0%7C637431294221313118%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=cl1ZZH47idYnki5wtVBgkE6IVA2%2BqqywwDzt284qcqk%3D&reserved=0> from Tim McMullan<mailto:mcmullan@schedmd.com> Hey Tom, Can you provide an example of a job submission that doesn't behave as expected? I'm interested in what options are getting passed in so I can properly replicate the situation! I did notice "CR_Pack_Nodes" which should (and so far in my testing seems to work) compact tasks like that if the tasks being launched don't consume the whole allocation. Thanks! --Tim ________________________________ You are receiving this mail because: * You reported the bug. We found our issue. Last September we changed SelectType=select/cons_res to SelectType=select/cons_tres And cons_tres doesn’t do the packing like it was. Put it back to cons_res and it works as it did. Is this expected behavior? Thanks! From: bugs@schedmd.com <bugs@schedmd.com> Sent: Wednesday, December 9, 2020 11:50 AM To: Tom Wurgler <twurgl@goodyear.com> Subject: [EXT] [Bug 10391] Packing nodes External Email....WARNING....Think before you click or respond....WARNING Comment # 1<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.schedmd.com%2Fshow_bug.cgi%3Fid%3D10391%23c1&data=04%7C01%7Ctwurgl%40goodyear.com%7Cf9ea50e3731845254dff08d89c628424%7C939e896692854a9a9f040887efe8aae0%7C0%7C0%7C637431294221313118%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=MmY3%2BJascwfJanoBAMv%2BSD2kotwrqiyc0CZYN9Xy3Jg%3D&reserved=0> on bug 10391<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.schedmd.com%2Fshow_bug.cgi%3Fid%3D10391&data=04%7C01%7Ctwurgl%40goodyear.com%7Cf9ea50e3731845254dff08d89c628424%7C939e896692854a9a9f040887efe8aae0%7C0%7C0%7C637431294221313118%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=cl1ZZH47idYnki5wtVBgkE6IVA2%2BqqywwDzt284qcqk%3D&reserved=0> from Tim McMullan<mailto:mcmullan@schedmd.com> Hey Tom, Can you provide an example of a job submission that doesn't behave as expected? I'm interested in what options are getting passed in so I can properly replicate the situation! I did notice "CR_Pack_Nodes" which should (and so far in my testing seems to work) compact tasks like that if the tasks being launched don't consume the whole allocation. Thanks! --Tim ________________________________ You are receiving this mail because: * You reported the bug. Hi Ok, We really want the cons-tres for use with gpu's. If we have tres, it splits jobs evenly across nodes If we have res, it fills a node then uses part of another node if needed. Then the next job fills that unused portion of the node. This "res" behavior is what we want, but we need "tres" for the gpu. What combination of settings can give us res packing with tres? This is a pretty big thing for us, so I raised the priority level. Thanks! tom We can have a teams meeting if we need to. Thanks Hey Tom, Sorry about the delay but I was out of the office at the end of the week so didn't make much progress there. I've been able to replicate and I'm currently examining the logic around this choice to see how exactly its making this decision, at which point I should have a better handle on it. Thanks! --Tim Hey Tom, Just wanted to update you on this - after looking through the code and chatting we've determined this is not the desired behavior. Right now, it doesn't appear that there is a good way to get the old allocation method back without code changes. I'm looking into some possible workarounds until a patch can be made. I'll update you with more when I can! Thanks! --Tim Hey Tom, I've been looking at this a lot and unfortunately there doesn't seem to be a clear way out here. I have a suggestion though that might get you similar behavior until a proper fix can be implemented. Can you try using "plane" distribution for now? EG: a job submission like "srun -m plane=12 --cpus-per-task=2 --ntasks=18" when allocating on empty 24 core nodes should result in 1 full node and 1 half-full node like it would in cons_res. Its worth noting here that the tasks might not be organized the same way as they would have been in cons_res. You can just specify something like "-m plane=24" so that regardless of the cpu/task count nodes should be filled before moving on to a different node. Can you try this out for a few of your jobs and let me know if it is a functional workaround for you? Thanks, --Tim Hey Tom, I just wanted to check in and see if you had a chance to look at the workaround I suggested! Thanks, --Tim I'm sorry, but not yet. Been off for 3 weeks and just ramping up again... Will this be fixed in a future version? If so, any idea when that version might come out? From: bugs@schedmd.com <bugs@schedmd.com> Sent: Tuesday, January 5, 2021 11:01 AM To: Tom Wurgler <twurgl@goodyear.com> Subject: [EXT] [Bug 10391] Packing nodes External Email....WARNING....Think before you click or respond....WARNING Comment # 11<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.schedmd.com%2Fshow_bug.cgi%3Fid%3D10391%23c11&data=04%7C01%7Ctwurgl%40goodyear.com%7C74014cdabc034d0e4bda08d8b1930ccd%7C939e896692854a9a9f040887efe8aae0%7C0%7C0%7C637454592434941000%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=qU9X59o%2BvnecgyU6spC3PuWcEb63lWAwFS6YZqRKloE%3D&reserved=0> on bug 10391<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.schedmd.com%2Fshow_bug.cgi%3Fid%3D10391&data=04%7C01%7Ctwurgl%40goodyear.com%7C74014cdabc034d0e4bda08d8b1930ccd%7C939e896692854a9a9f040887efe8aae0%7C0%7C0%7C637454592434951003%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=tAppj5Edj3UQY9Zs3c5h4UoT2ona%2BhTFccbHES1GYNY%3D&reserved=0> from Tim McMullan<mailto:mcmullan@schedmd.com> Hey Tom, I just wanted to check in and see if you had a chance to look at the workaround I suggested! Thanks, --Tim ________________________________ You are receiving this mail because: * You reported the bug. No worries, I hope you had a good vacation! We are looking at fixing this in a future release. I can't confirm when a fix would come out at the moment, but I will try to get that figured out for you. Thanks! --Tim I am back at looking at this. Sorry for the delay.
Using -m plane=24, I submitted 3 jobs. Note our cluster has 2 sockets, 12 cores per socket.
Job 1 - 8 core job
Job 2 - 8 core job
Job 3 - 32 core job
and we get this:
JOBID USER STATE PARTITION EXEC_HOST SUBMIT_TIME JOB_NAME
256840 t901353 RUNNING linlarge 8*rdsxen129 Feb 03 13:26 batch_script_7942.sh
256841 t901353 RUNNING linlarge 8*rdsxen129 Feb 03 13:28 batch_script_8799.sh
256842 t901353 RUNNING linlarge 7*rdsxen129:24*rdsxen130:1*rdsxen131
( 32 cpus) Feb 03 13:29 batch_script_9127.sh
Note that desired behavior is 8 on xen129, 8 more on xen129 and 8 more on xen129 before moving to xen130.
So no, -m plane isn't giving us what we want. So should I have put a different number than 24 in the plane=24?
And not to mention we have other clusters with different numbers of cores per node.
Any more thoughts here?
Is a potential fix in the source code being planned?
So I am not sure what we need out of con_tres for using gpus. Can we not take advantage of gpus if we dont' have cons_tres set? Can any gpu advanatages be used via the command line or possibly on a partition of gpu nodes and those advantages applied to the partition or per job?
Just trying to find a way around this issue.
thanks
tom
(In reply to Tom Wurgler from comment #15) > I am back at looking at this. Sorry for the delay. No worries, I'm glad you have been able to test the workaround! > Using -m plane=24, I submitted 3 jobs. Note our cluster has 2 sockets, 12 > cores per socket. > Job 1 - 8 core job > Job 2 - 8 core job > Job 3 - 32 core job > > and we get this: > JOBID USER STATE PARTITION EXEC_HOST SUBMIT_TIME > JOB_NAME > 256840 t901353 RUNNING linlarge 8*rdsxen129 Feb 03 13:26 > batch_script_7942.sh > 256841 t901353 RUNNING linlarge 8*rdsxen129 Feb 03 13:28 > batch_script_8799.sh > 256842 t901353 RUNNING linlarge 7*rdsxen129:24*rdsxen130:1*rdsxen131 > ( 32 cpus) Feb 03 13:29 > batch_script_9127.sh That's a strange result. I haven't been able to replicate that behavior locally, it seems to assign correctly for me. I'll try to see if there is something else going on. > Note that desired behavior is 8 on xen129, 8 more on xen129 and 8 more on > xen129 before moving to xen130. Totally understand that :) > So no, -m plane isn't giving us what we want. So should I have put a > different number than 24 in the plane=24? > And not to mention we have other clusters with different numbers of cores > per node. You can put an arbitrarily large number in plane=$count and it will split up based on the actual job/node size so you don't have to worry about it as much. Count could just be the largest number of CPUs you have in the cluster. > Any more thoughts here? If your above test is repeatable, it might be useful to have the slurmctld log with "SlurmctldDebug=debug2" and "DebugFlags=SelectType" in the slurm.conf and doing and scontrol reload. You can also enable those with: scontrol setdebug debug2 scontrol setdebugflags +SelectType We would at least need the logs from just before they were submitted to any time after they were scheduled. > Is a potential fix in the source code being planned? The ticket has been re-assigned to Dominik who is working in this code already. He is looking at the possible fixes. > So I am not sure what we need out of con_tres for using gpus. Can we not > take advantage of gpus if we dont' have cons_tres set? Can any gpu > advanatages be used via the command line or possibly on a partition of gpu > nodes and those advantages applied to the partition or per job? With cons_res, you can still schedule with GPUs, but its not as easy at job submit time. You can do something like "--gres=gpu:1" for a job and it will schedule it, and attempt to be smart about numa awareness if you've given it that information. --cpus-per-gpu, --gpus, --gpu-bind, --gpu-freq, --gpus-per-node, --gpus-per-socket, --gpus-per-task, --mem-per-gpu are all cons_tres specific. I'm not sure what your jobs GPU requirements are and how much of an issue this would or would not be, but its an option. Thanks! --Tim I redid my tests with "-m plane 40" Note I just made up this number. And thus far it is doing the correct thing. I am going to install the change to our standard script and watch how the packing goes. We are continuing to test.. thanks tom Adding the -m plane to the standard scripts still did not pack efficiently. thanks tom Hi Could you send us logs requested in comment 17? Dominik Created attachment 18048 [details]
log after setting the variables
scontrol setdebug debug2
scontrol setdebugflags +SelectType
Not exactly reproducible
l:t901353@rds4020:slurm5 > sjobs
JOBID USER STATE PARTITION EXEC_HOST SUBMIT_TIME JOB_NAME
257920 t901353 RUNNING linlarge 8*rdsxen65 Feb 22 08:56 sierra_batch_script_8363.sh
257921 t901353 RUNNING linlarge 8*rdsxen65 Feb 22 08:56 sierra_batch_script_8743.sh
257922 t901353 RUNNING linlarge 8*rdsxen65:24*rdsxen66
( 32 cpus) Feb 22 08:57 sierra_batch_script_8976.sh
257923 t901353 RUNNING linlarge 24*rdsxen259:8*rdsxen260
( 32 cpus) Feb 22 08:58 sierra_batch_script_9371.sh
257924 t901353 RUNNING linlarge 4*rdsxen324:4*rdsxen325:4*rdsxen326:4*rdsxen327:16*rdsxen328
( 32 cpus) Feb 22 08:58 sierra_batch_script_9612.sh
l:t901353@rds4020:slurm5 >
Still weird splitting
thanks
Hi Slurmctld spread out job 257924 over five nodes because it fills up partially allocated nodes rdsxen[324-327] as first. It looks strange, but this is correct behavior. I also have some updates about the primary issue of this bug. Unfortunately, the fix for it will be available only on 21.08. It will require a significant change in cons_tres, which we can't do in 20.11. Could we drop the severity of this bug to 3? Dominik It may be correct behavior, but overall this is not desirable behavior. This is still a big deal to us without having data to back it up. We think having jobs spread out like this will hurt performance. So I am going to run a series of tests to try to judge the effect of the current packing. As far as lowering priority, does the work on fixing it slow down then? From: bugs@schedmd.com <bugs@schedmd.com> Sent: Thursday, February 25, 2021 9:23 AM To: Tom Wurgler <twurgl@goodyear.com> Subject: [EXT] [Bug 10391] Packing nodes External Email....WARNING....Think before you click or respond....WARNING Comment # 26<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.schedmd.com%2Fshow_bug.cgi%3Fid%3D10391%23c26&data=04%7C01%7Ctwurgl%40goodyear.com%7C32672a0a50684614481e08d8d998eb5b%7C939e896692854a9a9f040887efe8aae0%7C0%7C0%7C637498598094001328%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=SsMWT%2FgPke6bsZTlvhlnAtWATqebnmAaFGN8OEqUWng%3D&reserved=0> on bug 10391<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.schedmd.com%2Fshow_bug.cgi%3Fid%3D10391&data=04%7C01%7Ctwurgl%40goodyear.com%7C32672a0a50684614481e08d8d998eb5b%7C939e896692854a9a9f040887efe8aae0%7C0%7C0%7C637498598094011316%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=9n0cjLPNF8SqWmf6NzlRhZQmSY%2BhRXn9XSQzzXZjKYM%3D&reserved=0> from Dominik Bartkiewicz<mailto:bart@schedmd.com> Hi Slurmctld spread out job 257924 over five nodes because it fills up partially allocated nodes rdsxen[324-327] as first. It looks strange, but this is correct behavior. I also have some updates about the primary issue of this bug. Unfortunately, the fix for it will be available only on 21.08. It will require a significant change in cons_tres, which we can't do in 20.11. Could we drop the severity of this bug to 3? Dominik ________________________________ You are receiving this mail because: * You reported the bug. Hi Topology-aware resource allocation minimizes the count of switches and simply takes nodes in order of bitmap position. Other flavors of allocation _eval_nodes_*() work differently. But generally, by default Slurm tries to use partially allocated nodes first. This behavior can be changed by LLN option. This fix for sure will be included in 21.09, and the severity level of this bug doesn't change this. Dominik Thanks for working on this. We look forward to the 21.8 version. I am lowering the priority back to 3. -tom Hi In 21.08 cons_tres should "pack" nodes as expected. https://github.com/SchedMD/slurm/compare/a3ff3d71cc20f...04f94c6a2229e6 Please reopen if anything else is found on this. Dominik *** Ticket 12605 has been marked as a duplicate of this ticket. *** |