Ticket 9865

Hi, is there any update on this issue, or do you need anything further from me?  I'd like to understand more about setting constraincores in cgroup.conf and if that might be of help for high load, excessive cpu usage situations like this.

Thanks,
Renata

Renata - I have Marcin looking into this for you. He will reply to you by tomorrow.

Thanks!

Renata

On Wed, 23 Sep 2020, bugs@schedmd.com wrote:

>https://bugs.schedmd.com/show_bug.cgi?id=9865
>
>--- Comment #2 from Jason Booth <jbooth@schedmd.com> ---
>Renata - I have Marcin looking into this for you. He will reply to you by
>tomorrow.
>
>-- 
>You are receiving this mail because:
>You reported the bug.

Renata,

If you are not constraining cores then it's possible to submit a single core job that will use more cores. The user may even be unaware of his disturbing activity since some applications/frameworks will for instance start as many tasks as many cores discovered by them.

I'd strongly recommend enabling the option in general. Is there any reaaon you decided to leave core unconstrained?

Cheers,
Marcin

Hi Marcin, to be honest, I think it was just an oversight coupled with
an incomplete understanding of what it did, even though the name
seems obvious enough.  I didn't actually think that the issue I
included in this ticket was a case of using more cores than requested
by the job.  The high load and slowness for the interactive users was
my main concern.  Can you tell anything about that from what I
included?  Is there a slurm cgroup (or other) setting that I should be
using that may address that?  And do I need to restart all the slurmds
to set constraincores, or just slurmctld?

Thanks,
Renata

On Wed, 23 Sep 2020, bugs@schedmd.com wrote:

>https://bugs.schedmd.com/show_bug.cgi?id=9865
>
>--- Comment #4 from Marcin Stolarek <cinek@schedmd.com> ---
>Renata,
>
>If you are not constraining cores then it's possible to submit a single core
>job that will use more cores. The user may even be unaware of his disturbing
>activity since some applications/frameworks will for instance start as many
>tasks as many cores discovered by them.
>
>I'd strongly recommend enabling the option in general. Is there any reaaon you
>decided to leave core unconstrained?
>
>Cheers,
>Marcin
>
>-- 
>You are receiving this mail because:
>You reported the bug.

Renata,

I may be wrong, since we just have an example job, but if the specification is consistent swmclau2 user specified NumNodes=1 NumCPUs=16, which means that for multiple threads his process should go up to 1600% of CPU, the top command shows ~26*100% which means that his running at least 26 threads.

>The high load and slowness for the interactive users was
my main concern. 
Yep, but this is probably caused by other jobs running on the cores slurmctld selected for those users.

>Is there a slurm cgroup (or other) setting that I should be
>using that may address that?  And do I need to restart all the slurmds
>to set constraincores, or just slurmctld?
Yes, change yor etc/slurm/cgroup.conf to have:
>ConstrainCores=Yes
and call `scontrol reconfigure`.

To verify if it works correctly go to the directory where you cgroups are mounted (by default it's /sys/fs/cgroup/) and check the content of cpuset/slurm/uid_UidOfUser/job_JobId/cpuset.cpus - it should contain a list of CPUs allowed like e.g "5-8".

This should be enough to limit job to use only CPUs it was given, however, the TaskPlugin stack we recommend (in general case) is:
>TaskPlugin=task/affinity, task/cgroup
This combination makes use of cgroups to limit CPUs/memory/devices accessible by a job and gives end-users additional --cpu-bind options implemented in task/affinity plugin.

Let me know if enabling cores constrainment worked as expected. Keep in mind that it will affect only new jobs, so you may need to wait some time for the changes to be fully implemented.

cheers,
Marcin

Hi Marcin, thanks for your analysis.  I have just implemented it and will see 
what happens.  Just to be sure, I did not have to systemctl restart slurmd
on the individual nodes, just restart slurmctld and scontrol reconfigure?

Renata

On Thu, 24 Sep 2020, bugs@schedmd.com wrote:

>https://bugs.schedmd.com/show_bug.cgi?id=9865
>
>--- Comment #6 from Marcin Stolarek <cinek@schedmd.com> ---
>Renata,
>
>I may be wrong, since we just have an example job, but if the specification is
>consistent swmclau2 user specified NumNodes=1 NumCPUs=16, which means that for
>multiple threads his process should go up to 1600% of CPU, the top command
>shows ~26*100% which means that his running at least 26 threads.
>
>>The high load and slowness for the interactive users was
>my main concern. 
>Yep, but this is probably caused by other jobs running on the cores slurmctld
>selected for those users.
>
>>Is there a slurm cgroup (or other) setting that I should be
>>using that may address that?  And do I need to restart all the slurmds
>>to set constraincores, or just slurmctld?
>Yes, change yor etc/slurm/cgroup.conf to have:
>>ConstrainCores=Yes
>and call `scontrol reconfigure`.
>
>To verify if it works correctly go to the directory where you cgroups are
>mounted (by default it's /sys/fs/cgroup/) and check the content of
>cpuset/slurm/uid_UidOfUser/job_JobId/cpuset.cpus - it should contain a list of
>CPUs allowed like e.g "5-8".
>
>This should be enough to limit job to use only CPUs it was given, however, the
>TaskPlugin stack we recommend (in general case) is:
>>TaskPlugin=task/affinity, task/cgroup
>This combination makes use of cgroups to limit CPUs/memory/devices accessible
>by a job and gives end-users additional --cpu-bind options implemented in
>task/affinity plugin.
>
>Let me know if enabling cores constrainment worked as expected. Keep in mind
>that it will affect only new jobs, so you may need to wait some time for the
>changes to be fully implemented.
>
>cheers,
>Marcin
>
>-- 
>You are receiving this mail because:
>You reported the bug.

>on the individual nodes, just restart slurmctld and scontrol reconfigure?
Just `scontrol reconfigure` was needed to get this change implemented, so you didn't have to bounce slurmctld.

Did you check if per user,job and step directories under cpuset filesystem are created and contain cpuste.cpus files with the content aligned with the result of scontrol show job -d JOBID? (I mean CPU indices displayed in the per node listing of the command)[1].

cheers,
Marcin

[1]JobId=36977 JobName=wrap
   UserId=root(0) GroupId=root(0) MCS_label=N/A
   Priority=66944 Nice=0 Account=root QOS=normal
   JobState=RUNNING Reason=None Dependency=(null)
   Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
   DerivedExitCode=0:0
   RunTime=00:00:02 TimeLimit=UNLIMITED TimeMin=N/A
   SubmitTime=2020-09-24T15:18:01 EligibleTime=2020-09-24T15:18:01
   AccrueTime=2020-09-24T15:18:01
   StartTime=2020-09-24T15:18:06 EndTime=Unknown Deadline=N/A
   PreemptEligibleTime=2020-09-24T15:18:06 PreemptTime=None
   SuspendTime=None SecsPreSuspend=0 LastSchedEval=2020-09-24T15:18:06
   Partition=AllNodes AllocNode:Sid=slurmctl:7785
   ReqNodeList=(null) ExcNodeList=(null)
   NodeList=test01
   BatchHost=test01
   NumNodes=1 NumCPUs=2 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
   TRES=cpu=2,node=1,billing=2
   Socks/Node=* NtasksPerN:B:S:C=1:0:*:* CoreSpec=*
   JOB_GRES=(null)
     Nodes=test01 CPU_IDs=0-1 Mem=0 GRES=
                   ^^^^^^^^^^^^-------------- List of CPUs assigned by slurmctld
   MinCPUsNode=1 MinMemoryNode=10M MinTmpDiskNode=0
   Features=(null) DelayBoot=00:00:00
   OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
   Command=(null)
   WorkDir=/slurm-sources/bug9851
   StdErr=/slurm-sources/bug9851/slurm-36977.out
   StdIn=/dev/null
   StdOut=/slurm-sources/bug9851/slurm-36977.out
   Power=
   MailUser=(null) MailType=NONE

Hi Marcin, I just ran a job requesting 6 cores and see this on the running host:

[root@rome0142 job_61192]# pwd
/sys/fs/cgroup/cpuset/slurm/uid_1197/job_61192
[root@rome0142 job_61192]# cat cpuset.cpus
0-2,64-66

and the same under step_0 and step_batch.

Renata


On Thu, 24 Sep 2020, bugs@schedmd.com wrote:

>https://bugs.schedmd.com/show_bug.cgi?id=9865
>
>--- Comment #8 from Marcin Stolarek <cinek@schedmd.com> ---
>>on the individual nodes, just restart slurmctld and scontrol reconfigure?
>Just `scontrol reconfigure` was needed to get this change implemented, so you
>didn't have to bounce slurmctld.
>
>Did you check if per user,job and step directories under cpuset filesystem are
>created and contain cpuste.cpus files with the content aligned with the result
>of scontrol show job -d JOBID? (I mean CPU indices displayed in the per node
>listing of the command)[1].
>
>cheers,
>Marcin
>
>[1]JobId=36977 JobName=wrap
>   UserId=root(0) GroupId=root(0) MCS_label=N/A
>   Priority=66944 Nice=0 Account=root QOS=normal
>   JobState=RUNNING Reason=None Dependency=(null)
>   Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
>   DerivedExitCode=0:0
>   RunTime=00:00:02 TimeLimit=UNLIMITED TimeMin=N/A
>   SubmitTime=2020-09-24T15:18:01 EligibleTime=2020-09-24T15:18:01
>   AccrueTime=2020-09-24T15:18:01
>   StartTime=2020-09-24T15:18:06 EndTime=Unknown Deadline=N/A
>   PreemptEligibleTime=2020-09-24T15:18:06 PreemptTime=None
>   SuspendTime=None SecsPreSuspend=0 LastSchedEval=2020-09-24T15:18:06
>   Partition=AllNodes AllocNode:Sid=slurmctl:7785
>   ReqNodeList=(null) ExcNodeList=(null)
>   NodeList=test01
>   BatchHost=test01
>   NumNodes=1 NumCPUs=2 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
>   TRES=cpu=2,node=1,billing=2
>   Socks/Node=* NtasksPerN:B:S:C=1:0:*:* CoreSpec=*
>   JOB_GRES=(null)
>     Nodes=test01 CPU_IDs=0-1 Mem=0 GRES=
>                   ^^^^^^^^^^^^-------------- List of CPUs assigned by
>slurmctld
>   MinCPUsNode=1 MinMemoryNode=10M MinTmpDiskNode=0
>   Features=(null) DelayBoot=00:00:00
>   OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
>   Command=(null)
>   WorkDir=/slurm-sources/bug9851
>   StdErr=/slurm-sources/bug9851/slurm-36977.out
>   StdIn=/dev/null
>   StdOut=/slurm-sources/bug9851/slurm-36977.out
>   Power=
>   MailUser=(null) MailType=NONE
>
>-- 
>You are receiving this mail because:
>You reported the bug.

Looks good, let's check with interactive users when after MaxTime for the partition in question(shared) or other partitions overlaping with it.

cheers,
Marcin

Hi Marcin, we are still seeing high load on the hosts with jobs running
from user swmclau2.  We think it is related to the number of threads
his jobs are using.  His userid is  14185:

[renata@rome0122 ~]$ squeue | grep rome0122
         62173_751    shared  trainer swmclau2  R    2:35:49      1 rome0122 
         62173_752    shared  trainer swmclau2  R    2:35:49      1 rome0122 
         62173_756    shared  trainer swmclau2  R    2:35:49      1 rome0122 
         62173_758    shared  trainer swmclau2  R    2:35:49      1 rome0122 
         62173_761    shared  trainer swmclau2  R    2:35:49      1 rome0122 
         62173_762    shared  trainer swmclau2  R    2:35:49      1 rome0122 
         62173_772    shared  trainer swmclau2  R    2:35:49      1 rome0122 
         62173_773    shared  trainer swmclau2  R    2:35:49      1 rome0122 

[renata@rome0122 ~]$ w
 11:16:40 up 12 days, 20:07,  1 user,  load average: 1009.70, 799.51, 722.91
USER     TTY      FROM             LOGIN@   IDLE   JCPU   PCPU WHAT
renata   pts/0    sdf-login01.slac 11:16    0.00s  0.21s  0.03s w

[renata@rome0122 ~]$ ps axms | grep ^14185 | grep -c ' Rl'
1024

As the number of threads he uses drops then so does the load.

I am wondering if the numa stuff is set up properly.  The nodes are amd epyc 7702
systems and I have

SchedulerParameters=Ignore_NUMA

We haven't turned off the numa balancing in the OS though.  The nodes
look like:

[renata@rome0001 ~]$ lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                128
On-line CPU(s) list:   0-127
Thread(s) per core:    1
Core(s) per socket:    64
Socket(s):             2
NUMA node(s):          8
Vendor ID:             AuthenticAMD
CPU family:            23
Model:                 49
Model name:            AMD EPYC 7702 64-Core Processor
Stepping:              0
CPU MHz:               1996.252
BogoMIPS:              3992.50
Virtualization:        AMD-V
L1d cache:             32K
L1i cache:             32K
L2 cache:              512K
L3 cache:              16384K
NUMA node0 CPU(s):     0-15
NUMA node1 CPU(s):     16-31
NUMA node2 CPU(s):     32-47
NUMA node3 CPU(s):     48-63
NUMA node4 CPU(s):     64-79
NUMA node5 CPU(s):     80-95
NUMA node6 CPU(s):     96-111
NUMA node7 CPU(s):     112-127
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb cat_l3 cdp_l3 hw_pstate sme retpoline_amd ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif umip overflow_recov succor smca
[renata@rome0001 ~]$ 

Does this seem like the right set-up?

Thanks,
Renata


On Fri, 25 Sep 2020, bugs@schedmd.com wrote:

>https://bugs.schedmd.com/show_bug.cgi?id=9865
>
>--- Comment #10 from Marcin Stolarek <cinek@schedmd.com> ---
>Looks good, let's check with interactive users when after MaxTime for the
>partition in question(shared) or other partitions overlaping with it.
>
>cheers,
>Marcin
>
>-- 
>You are receiving this mail because:
>You reported the bug.

Renata,

>we are still seeing high load on the hosts with jobs running
>from user swmclau2.  We think it is related to the number of threads
>his jobs are using.

That's actually expected and you're correct it's related to number of threads/processes running on the host. Per load definition it's number of processes in running or ready queue (waiting or using CPU resources). However, if specific processes are bound to appropriate cores high load should not imapct all CPUs, since other jobs are assigned theirs individual resources and are not competing with the user threads. Do you see "slowness" from the interactive job running on the same host as the highly threaded one?

>SchedulerParameters=Ignore_NUMA
Setting this for cluster with AMD Epyc sounds resonable.

cheers,
Marcin

Hi Marcin, there were no interactive jobs running this time.
And when I logged in, the system seemed responsive.  I'll
monitor some more and see try to check with any interactive
users that I see running on the same host.

Thanks,
Renata

  

On Tue, 29 Sep 2020, bugs@schedmd.com wrote:

>https://bugs.schedmd.com/show_bug.cgi?id=9865
>
>--- Comment #12 from Marcin Stolarek <cinek@schedmd.com> ---
>Renata,
>
>>we are still seeing high load on the hosts with jobs running
>>from user swmclau2.  We think it is related to the number of threads
>>his jobs are using.
>
>That's actually expected and you're correct it's related to number of
>threads/processes running on the host. Per load definition it's number of
>processes in running or ready queue (waiting or using CPU resources). However,
>if specific processes are bound to appropriate cores high load should not
>imapct all CPUs, since other jobs are assigned theirs individual resources and
>are not competing with the user threads. Do you see "slowness" from the
>interactive job running on the same host as the highly threaded one?
>
>>SchedulerParameters=Ignore_NUMA
>Setting this for cluster with AMD Epyc sounds resonable.
>
>cheers,
>Marcin
>
>-- 
>You are receiving this mail because:
>You reported the bug.

>And when I logged in, the system seemed responsive.

This is a good symptom though the load is high it doesn't impact all processes, since the set of threads responsible for that is limited to appropriate CPUs.

> I'll monitor some more and see try to check with any interactive users that I see running on the same host.

It should be all right. The "system responsiveness" verification you did is essentially the same thing.

cheers,
Marcin

 Renata,

Did you have a chance to verify the solution - either logging into nodes with higher load or checking interactive users' experience?

cheers,
Marcin

Hi Marcin, I haven't been able to get anything definitive out of
users, but I haven't heard anymore complaints either.  I continue to
see high loads on the hosts running jobs for that one user, but they
are also very responsive.  I think we should say that setting
constraincores has helped/fixed the issue and I'll open up a new
ticket if needed.

Thanks for all of your help,
Renata

On Wed, 7 Oct 2020, bugs@schedmd.com wrote:

>https://bugs.schedmd.com/show_bug.cgi?id=9865
>
>--- Comment #15 from Marcin Stolarek <cinek@schedmd.com> ---
> Renata,
>
>Did you have a chance to verify the solution - either logging into nodes with
>higher load or checking interactive users' experience?
>
>cheers,
>Marcin
>
>-- 
>You are receiving this mail because:
>You reported the bug.