Ticket 13000

it can also start immediately, without the wait.

Please attach your slurm.conf, cgroup.conf. I do see your config snippets in the other bug, however, we would like to review how you have Slurm setup.

Konstantin,

If the allocation is running on the job node then I believe that "TaskPlugin=task/cgroup,task/affinity" will still fix your issue.

If you don't have "LaunchParameters=use_interactive_step" in your slurm.conf you may be running on the login node and you would need to launch a step with srun to do something on the actual compute node.

There are other configuration issues that may be causing an issue like this. As Jason mentioned, please attach your slurm.conf, and cgroup.conf it helps us a lot in diagnosing issues.

Also our ticket severity is based on impact to production. A High Impact issue is described as "a problem that is causing sporadic outages or is consistently encountered by end users with adverse impact to end user interaction with the system." Is this issue causing this type of problem on your production system? It seems more like a level 3 or 4 to me.
https://www.schedmd.com/support.php

-Scott

Hello,

I'm a colleague of Konstantin.

We found the cullprit and it is not related to slurm at all. Here is the story that could be useful for others.

We are using a weka filesystem. This filesystem is running an agent that is simply resetting (!) the cpuset every 5 minutes by default. This is configurable in the /etc/wekaio/service.conf file by setting isolate_cpusets to false. You need weka version 3.11.3+ to make this work, previous versions are subject to a bug. 

There is an entry in their knownledge base about this, but you need a valid login. 

https://support.weka.io/s/article/Using-Slurm-or-another-job-scheduler-with-Weka

Here is a copy/paste of the interesting section which is quite generic, nothing secret about this:

=============================
If using cpuset within Slurm to pin user jobs to certain cores/memory, it is important to make sure Weka does not modify
these settings. By default Weka will reset CPUsets when the Weka agent is running.

To ensure Weka does not modify the cpusets, you must change the file "/etc/wekaio/service.conf".
Look for the line:

isolate_cpusets=true

Change this to:

isolate_cpusets=false


Then restart the weka agent:

systemctl restart weka-agent
=============================

Again, you'll need at least >= weka 3.13.3 on the client to make it work.

> A High Impact issue is described as "a problem that is causing sporadic outages 
> or is consistently encountered by end users with adverse impact to end user 
> interaction with the system." Is this issue causing this type of problem on your
> production system?

Yes it has an adverse impact, since any users can access all compute ressources from any node and all jobs were not ressources constrained at all after a few minutes run.

Sorry about opening two tickets too soon, but we were absolutly clueless at the time. 

Feel free to close this case. It should also close #12990.

Jean-Baptiste

*** Ticket 12990 has been marked as a duplicate of this ticket. ***

Jean-Baptiste,

Thanks for the info. I am glad you got it figured out.

-Scott

Hello all,
I have a question here. Does this behavior happen only with "sacct"? Or it appears with batch jobs as well?
I can replicate this with "sacct" on our side
Best regards,
Ahmed

Hello Ahmed,

The problem is independent of slurm. Any process inside a cgroup on a weka client with isolate_cpusets=true (the default) will escape after a minutes.

Jean-Baptiste

Dear Jean-Baptiste,
Thanks a lot for the helpful information.
Just to avoid confusion, in comment 5 you mentioned:
> You need weka version 3.11.3+ to make this work, previous versions are subject to a bug.

But also mentioned a different version here:
> Again, you'll need at least >= weka 3.13.3 on the client to make it work.

Can you please confirm the Weka client version that has the fix?

Thanks again
Ahmed

You're right it's 3.11.3+ ! Sorry about that.

Glad we could help.