Ticket 671

Could please upload the slurmd log file?
Do you use the release agent provided with Slurm?


On 03/31/2014 12:45 PM, bugs@schedmd.com wrote:
> Site 	CRAY
> Bug ID 	671 <http://bugs.schedmd.com/show_bug.cgi?id=671>
> Summary 	sbatch error: unable to remove step memcg
> Product 	SLURM
> Version 	14.11.x
> Hardware 	Linux
> OS 	Linux
> Status 	UNCONFIRMED
> Severity 	4 - Minor Issue
> Priority 	---
> Component 	slurmd daemon
> Assignee 	david@schedmd.com
> Reporter 	dgloe@cray.com
> CC 	da@schedmd.com, david@schedmd.com, jette@schedmd.com
>
> On sbatch jobs, we're consistently seeing this error at the end of job output:
> slurmstepd: task/cgroup: unable to remove step memcg : Device or resource busy
>
> This results in leftover memory control groups with no tasks like this one:
> # cat /dev/mcgroup/slurm/uid_29597/job_7045/step_4294967294/tasks
> #
>
> There must be some process left in the memory control group when it's being
> removed.
>
> We're using TaskPlugin=task/affinity,task/cgroup,task/cray.
>
> ------------------------------------------------------------------------
> You are receiving this mail because:
>
>   * You are on the CC list for the bug.
>   * You are the assignee for the bug.
>   * You are watching someone on the CC list of the bug.
>   * You are watching the assignee of the bug.
>

Created attachment 723 [details]
slurmd log file showing the problem

We don't use the Slurm release agent. Here's our cgroup.conf:

###
#
# Slurm cgroup support configuration file
#
# See man slurm.conf and man cgroup.conf for further
# information on cgroup configuration parameters
#--
CgroupAutomount=yes
#CgroupReleaseAgentDir="/etc/slurm/cgroup"
CgroupMountpoint="/dev"
ConstrainCores=yes
ConstrainRAMSpace=yes
TaskAffinity=no

Thanks, I am investigating the problem.

On 03/31/2014 01:35 PM, bugs@schedmd.com wrote:
> *Comment # 2 <http://bugs.schedmd.com/show_bug.cgi?id=671#c2> on bug 671
> <http://bugs.schedmd.com/show_bug.cgi?id=671> from David Gloe
> <mailto:dgloe@cray.com> *
>
> Createdattachment 723  <attachment.cgi?id=723>  [details]  <attachment.cgi?id=723&action=edit>
> slurmd log file showing the problem
>
> We don't use the Slurm release agent. Here's our cgroup.conf:
>
> ###
> #
> # Slurm cgroup support configuration file
> #
> # See man slurm.conf and man cgroup.conf for further
> # information on cgroup configuration parameters
> #--
> CgroupAutomount=yes
> #CgroupReleaseAgentDir="/etc/slurm/cgroup"
> CgroupMountpoint="/dev"
> ConstrainCores=yes
> ConstrainRAMSpace=yes
> TaskAffinity=no
>
> ------------------------------------------------------------------------
> You are receiving this mail because:
>
>   * You are on the CC list for the bug.
>   * You are the assignee for the bug.
>   * You are watching someone on the CC list of the bug.
>   * You are watching the assignee of the bug.
>

The messages you seen indicate that there are still processes using the cgroup,
or jobs belonging to the same user running on the machine. The release agent
provided with Slurm should clean those directories after the last process
exit. Why are you not use the release agent?

David

(In reply to David Bigagli from comment #4)
> The messages you seen indicate that there are still processes using the
> cgroup,
> or jobs belonging to the same user running on the machine. The release agent
> provided with Slurm should clean those directories after the last process
> exit. Why are you not use the release agent?
> 
> David

We don't use the Slurm release agent due to some quirkiness on our compute nodes. Slurm is installed in a chroot environment, along with its release agent, so the release agent path (which is based on the actual root, not the chroot) isn't set correctly in the cgroup by Slurm.

We do have our own release agent we use for cpusets; I'll try setting the mcgroup release agent to that as well and see if that helps at all.

One think we do in our release agent is to lock, using the flock 
command, the entire subsystem before doing anything, since the code that 
creates the cgroups uses the flock() sys call as well this allows for a 
mutual exclusion.

On 04/02/2014 10:55 AM, bugs@schedmd.com wrote:
> *Comment # 5 <http://bugs.schedmd.com/show_bug.cgi?id=671#c5> on bug 671
> <http://bugs.schedmd.com/show_bug.cgi?id=671> from David Gloe
> <mailto:dgloe@cray.com> *
>
> (In reply to David Bigagli fromcomment #4  <show_bug.cgi?id=671#c4>)
>> The messages you seen indicate that there are still processes using the
>> cgroup,
>> or jobs belonging to the same user running on the machine. The release agent
>> provided with Slurm should clean those directories after the last process
>> exit. Why are you not use the release agent?
>>
>> David
>
> We don't use the Slurm release agent due to some quirkiness on our compute
> nodes. Slurm is installed in a chroot environment, along with its release
> agent, so the release agent path (which is based on the actual root, not the
> chroot) isn't set correctly in the cgroup by Slurm.
>
> We do have our own release agent we use for cpusets; I'll try setting the
> mcgroup release agent to that as well and see if that helps at all.
>
> ------------------------------------------------------------------------
> You are receiving this mail because:
>
>   * You are on the CC list for the bug.
>   * You are the assignee for the bug.
>   * You are watching someone on the CC list of the bug.
>   * You are watching the assignee of the bug.
>

I've dug up a potential cause:
http://linux-kernel.2935.n7.nabble.com/PATCH-cgroup-fix-rmdir-EBUSY-regression-in-3-11-td710818.html

I'm looking to see if we have that patch on our compute nodes.

David, do you have an update on this issue?

Thanks, David

After some more investigation, this is definitely a Cray kernel issue. I don't believe Slurm is at fault unless it's doing something strange that results in the memory control group having memory usage remaining when no tasks are left.