Hi! this seem to be more general issue than in https://bugs.schedmd.com/show_bug.cgi?id=12990 so I open a different ticket. to reproduce: fill a node with user jobs, leave one core. salloc that core. (0)-(user@maestro-submit:~)-> salloc salloc: Pending job allocation 53507687 <snip> salloc: Nodes maestro-1041 are ready for job Our default is 1 core and 4Gb run: sysbench cpu --threads=90 --time=100000 run sysbench 1.0.20 (using system LuaJIT 2.1.0-beta3) Running the test with following options: Number of threads: 90 <snip> Threads started! now do: (0)-(user@maestro-1041:~)-> ps -L -C sysbench --no-headers -o psr |uniq 8 which is great, all threads on that one allocated core. but wait about 5 minutes, and we have: (0)-(user@maestro-1041:~)-> ps -L -C sysbench --no-headers -o psr |uniq 8 13 23 20 21 38 <snip> (0)-(user@maestro-1041:~)-> ps -L -C sysbench --no-headers -o psr |uniq |wc -l 90 so basically salloc doesn't limit anything, anybody can take all cores on a system. there seems to be nothing about this subject in your 21.x release log. please advice. best, Konstantin
it can also start immediately, without the wait.
Please attach your slurm.conf, cgroup.conf. I do see your config snippets in the other bug, however, we would like to review how you have Slurm setup.
Konstantin, If the allocation is running on the job node then I believe that "TaskPlugin=task/cgroup,task/affinity" will still fix your issue. If you don't have "LaunchParameters=use_interactive_step" in your slurm.conf you may be running on the login node and you would need to launch a step with srun to do something on the actual compute node. There are other configuration issues that may be causing an issue like this. As Jason mentioned, please attach your slurm.conf, and cgroup.conf it helps us a lot in diagnosing issues. Also our ticket severity is based on impact to production. A High Impact issue is described as "a problem that is causing sporadic outages or is consistently encountered by end users with adverse impact to end user interaction with the system." Is this issue causing this type of problem on your production system? It seems more like a level 3 or 4 to me. https://www.schedmd.com/support.php -Scott
Hello, I'm a colleague of Konstantin. We found the cullprit and it is not related to slurm at all. Here is the story that could be useful for others. We are using a weka filesystem. This filesystem is running an agent that is simply resetting (!) the cpuset every 5 minutes by default. This is configurable in the /etc/wekaio/service.conf file by setting isolate_cpusets to false. You need weka version 3.11.3+ to make this work, previous versions are subject to a bug. There is an entry in their knownledge base about this, but you need a valid login. https://support.weka.io/s/article/Using-Slurm-or-another-job-scheduler-with-Weka Here is a copy/paste of the interesting section which is quite generic, nothing secret about this: ============================= If using cpuset within Slurm to pin user jobs to certain cores/memory, it is important to make sure Weka does not modify these settings. By default Weka will reset CPUsets when the Weka agent is running. To ensure Weka does not modify the cpusets, you must change the file "/etc/wekaio/service.conf". Look for the line: isolate_cpusets=true Change this to: isolate_cpusets=false Then restart the weka agent: systemctl restart weka-agent ============================= Again, you'll need at least >= weka 3.13.3 on the client to make it work. > A High Impact issue is described as "a problem that is causing sporadic outages > or is consistently encountered by end users with adverse impact to end user > interaction with the system." Is this issue causing this type of problem on your > production system? Yes it has an adverse impact, since any users can access all compute ressources from any node and all jobs were not ressources constrained at all after a few minutes run. Sorry about opening two tickets too soon, but we were absolutly clueless at the time. Feel free to close this case. It should also close #12990. Jean-Baptiste
*** Ticket 12990 has been marked as a duplicate of this ticket. ***
Jean-Baptiste, Thanks for the info. I am glad you got it figured out. -Scott
Hello all, I have a question here. Does this behavior happen only with "sacct"? Or it appears with batch jobs as well? I can replicate this with "sacct" on our side Best regards, Ahmed
Hello Ahmed, The problem is independent of slurm. Any process inside a cgroup on a weka client with isolate_cpusets=true (the default) will escape after a minutes. Jean-Baptiste
Dear Jean-Baptiste, Thanks a lot for the helpful information. Just to avoid confusion, in comment 5 you mentioned: > You need weka version 3.11.3+ to make this work, previous versions are subject to a bug. But also mentioned a different version here: > Again, you'll need at least >= weka 3.13.3 on the client to make it work. Can you please confirm the Weka client version that has the fix? Thanks again Ahmed
You're right it's 3.11.3+ ! Sorry about that. Glad we could help.
