Hi, It looks like we have some interactive jobs failing because of slurmstepd segfaults. I can't find any core file on the compute nodes, though. We do have messages like: slurmstepd[40841]: segfault at 0 ip 00000030a1d2a7a5 sp 00007fd2551818f8 error 4 in libc-2.12.so[30a1c00000+18b000] It seems to have started about when we enabled ConstraintDevices in cgroup.conf, and seems to be triggered by the following kind of job: $ salloc srun --pty bash I'd love to provide some more information (esp. core file) but I'm not sure why they're not generated. Thanks.
Hi, the core files are in the log directory. If they are not generated perhaps the slurmd is started with ulimit 0 for core files? ulimit -c. Another possibility is that there is a global limit imposed on core file creation in /etc/security/limits.conf ? David
(In reply to David Bigagli from comment #1) > Hi, the core files are in the log directory. If they are not generated > perhaps > the slurmd is started with ulimit 0 for core files? ulimit -c. > Another possibility is that there is a global limit imposed on core file > creation in /etc/security/limits.conf ? I checked that this is not the case. In that specific case, PID of slurmstepd is 43563. And core size is not limited: sh-5-33 # grep core /proc/43563/limits Max core file size unlimited unlimited bytes Yet, when it segfaults, there's no core in the log directory: sh-5-33 # dmesg | grep 43563 slurmstepd[43563]: segfault at 0 ip 0000003d3512a7a5 sp 00007f7ee708d8f8 error 4 in libc-2.12.so[3d35000000+18b000] sh-5-33 # ls -al /var/log/slurm/ total 380 drwxr-xr-x 2 daemon daemon 4096 Aug 5 16:13 . drwxr-xr-x 17 root root 4096 Sep 21 03:50 .. -rw------- 1 root root 375938 Sep 24 16:37 slurmd.log
Do you have an exact reproducer or it is random? If it is not just attach a gdb and have it core dump under you? Did you change JobAcctGatherType by any chance? This changes the format of the protocol and cannot be done when steps are around? What is in the slurmd log file? David
(In reply to David Bigagli from comment #3) > Do you have an exact reproducer or it is random? It seems to be pretty easy to reproduce with: $ salloc srun --pty bash Afetr a few seconds, slurmstepd segfaults the session freezes. > If it is not just attach a > gdb > and have it core dump under you? Yep, I just did that. Here's the backtrace: (gdb) bt #0 0x0000003d3512a7a5 in __strstr_sse42 () from /lib64/libc.so.6 #1 0x00007fccf0f09411 in _prec_extra (prec=0x7fccd4004bc8) at jobacct_gather_cgroup.c:128 #2 0x00007fccf0f0b1c9 in _handle_stats (prec_list=0x1259918, proc_stat_file=0x7fccf0f05b10 "/proc/3026/stat", proc_io_file=0x7fccf0f05a10 "/proc/3026/io", callbacks=0x7fccf110d900) at common_jag.c:407 #3 0x00007fccf0f0b70d in _get_precs (task_list=<value optimized out>, pgid_plugin=<value optimized out>, cont_id=<value optimized out>, callbacks=0x7fccf110d900) at common_jag.c:447 #4 0x00007fccf0f0b7be in jag_common_poll_data (task_list=0x12595f8, pgid_plugin=<value optimized out>, cont_id=<value optimized out>, callbacks=0x7fccf110d900) at common_jag.c:610 #5 0x00000000004c57c3 in _poll_data () at slurm_jobacct_gather.c:204 #6 0x00000000004c5bc4 in _watch_tasks (arg=<value optimized out>) at slurm_jobacct_gather.c:231 #7 0x0000003d358079d1 in start_thread () from /lib64/libpthr > Did you change JobAcctGatherType by any chance? This changes the format of > the protocol and cannot be done when steps are around? I did, I'm trying to use the jobacct_gather/cgroup plugin. But the segfaults also occur on a test node where no previous jobs nor steps were running. > What is in the slurmd log file? For a job that makes slurmstepd segfault: [2014-09-24T17:23:33.565] launch task 360674.0 request from 215845.32264@10.210.31.251 (port 26600) [2014-09-24T17:23:33.591] [360674.0] can't stat gres.conf file /etc/slurm/gres.conf, assuming zero resource counts [2014-09-24T17:23:33.591] [360674.0] checkpoint/blcr init [2014-09-24T17:23:33.606] [360674.0] task/cgroup: /slurm/uid_215845/job_360674: alloc=0MB mem.limit=64380MB memsw.limit=64380MB [2014-09-24T17:23:33.606] [360674.0] task/cgroup: /slurm/uid_215845/job_360674/step_0: alloc=0MB mem.limit=64380MB memsw.limit=64380MB [2014-09-24T17:23:33.606] [360674.0] task/cgroup: manage devices jor job '360674' [2014-09-24T17:23:33.606] [360674.0] can't stat gres.conf file /etc/slurm/gres.conf: No such file or directory [2014-09-24T17:23:33.607] [360674.0] Default access allowed to device c 1:3 rwm [... lot of other similar lines for other devices ...] [2014-09-24T17:23:33.609] [360674.0] Default access allowed to device c 4:4 rwm [2014-09-24T17:23:33.614] [360674.0] in _window_manager Thanks!
Hi Kilian, I can reproduce this as well if I configure: JobAcctGatherType=jobacct_gather/cgroup the code appears to access a path that does not exist and it goes downhill from there. I also don't have any core even if cwd is the log directory as expected. I am still investigating, if you set JobAcctGatherType=jobacct_gather/linux you should be ok for now. David
Hi David, (In reply to David Bigagli from comment #5) > Hi Kilian, > I can reproduce this as well if I configure: > > JobAcctGatherType=jobacct_gather/cgroup Excellent, I'm glad it's reproducible. Is it related to ConstrainDevices, or does it also happens when ConstrainDevices is off? > the code appears to access a path that does not exist and it goes downhill > from there. I also don't have any core even if cwd is the log directory > as expected. > > I am still investigating, Great, thanks. > if you set > > JobAcctGatherType=jobacct_gather/linux > > you should be ok for now. Yes, that's exactly what I did for now and it seems that I don't get segfaults anymore. Thanks a lot!
I just tried without ConstrainDevices and it made no difference. Even more interesting that it seems to work on Ubuntu 14.04.1 ->uname -a Linux smd-server 3.13.0-35-generic #62-Ubuntu SMP Fri Aug 15 01:58:42 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux but not on: ->uname -a;cat /etc/redhat-release Linux prometeo 2.6.32-431.29.2.el6.x86_64 #1 SMP Tue Sep 9 21:36:05 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux CentOS release 6.5 (Final) Oh well... David
Interesting news. What happens is as follow. When starting the job we create the hierarchy in /cgroup, slurmstepd builds the full path: /cgroup/memory/slurm_dario/uid_500/job_105661/step_0/task_0 and creates the directories. It is possible that some of these directories exist already as other plugins have already created it as they use cgroup as well, for example the task plugin. Now *sometimes* in the memory cgroup if the path /cgroup/memory/slurm_dario/uid_500/job_105661/step_0 exist already the last mkdir(task_0) fails, but sometimes does not.. Would you be able to run an instrumented code in which we watch for the mkdir() to fail catch it and then prevent slurmstepd from dying? David
(In reply to David Bigagli from comment #8) > Interesting news. What happens is as follow. When starting the job we create > the hierarchy in /cgroup, slurmstepd builds the full path: > > /cgroup/memory/slurm_dario/uid_500/job_105661/step_0/task_0 > > and creates the directories. It is possible that some of these directories > exist already as other plugins have already created it as they use cgroup as > well, > for example the task plugin. Now *sometimes* in the memory cgroup if the path > /cgroup/memory/slurm_dario/uid_500/job_105661/step_0 exist already the last > mkdir(task_0) fails, but sometimes does not.. Wow, that's easy... > Would you be able to run an instrumented code in which we watch for the > mkdir() to fail catch it and then prevent slurmstepd from dying? Sure! As long as it only involves one compute node, and that the rest of the cluster can continue to work, I'd be glad to do so.
When cgroups are created before the job start the slurmstepd creates the hierarchy by mkdir(). It appears that sometimes mkdir() silently fails and that's why part of the hierarchy is not created. The accounting plugin blindly access the path the does not exists and croaks. This should be fixed of course. What is interesting is that if I slow the code down, either using gdb or by printing detailed steps the mkdir() works and creates the hierarchy correctly. I have instrumented the code for you to try on the test node. You will see this sort of print: Sep 25 14:17:30.548424 12844 0x7ff49a256700] [105671.0] xcgroup_instanciate: mkdir(/cgroup/memory/slurm_dario/uid_500/job_105671/step_0/task_0) [Sep 25 14:17:30.590141 12844 0x7ff49a256700] [105671.0] xcgroup_instanciate: done mkdir(/cgroup/memory/slurm_dario/uid_500/job_105671/step_0/task_0) cc 0 I assume it will be ok on your install as well. Can you also append your uname -a? David
Created attachment 1261 [details] cgroup instrumentation
->test node where no previous jobs nor steps were running. Just to make sure I understand this right. You have a dedicated node for testing as a standalone cluster? Or your node is part of the rest of the cluster and you use it as a testing machine. In the first case the instrumentation is fine. In the second case it will not work since we modified the plugin and not a daemon only. We cannot have a node in the cluster running with different plugins. David
(In reply to David Bigagli from comment #10) > When cgroups are created before the job start the slurmstepd creates the > hierarchy by mkdir(). It appears that sometimes mkdir() silently fails > and that's why part of the hierarchy is not created. The accounting plugin > blindly access the path the does not exists and croaks. This should be fixed > of course. Ok, that makes sense, thanks for the explanation. > What is interesting is that if I slow the code down, either using gdb or by > printing detailed steps the mkdir() works and creates the hierarchy > correctly. Wow, that's weird. A race condition maybe? > I have instrumented the code for you to try on the test node. > > You will see this sort of print: > > Sep 25 14:17:30.548424 12844 0x7ff49a256700] [105671.0] xcgroup_instanciate: > mkdir(/cgroup/memory/slurm_dario/uid_500/job_105671/step_0/task_0) > [Sep 25 14:17:30.590141 12844 0x7ff49a256700] [105671.0] > xcgroup_instanciate: done > mkdir(/cgroup/memory/slurm_dario/uid_500/job_105671/step_0/task_0) cc 0 > > I assume it will be ok on your install as well. > Can you also append your uname -a? Linux sh-5-33.local 2.6.32-431.23.3.el6.x86_64 #1 SMP Wed Jul 16 06:12:23 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux That's RHEL 6.5 Thanks!
(In reply to David Bigagli from comment #12) > > ->test node where no previous jobs nor steps were running. > > Just to make sure I understand this right. You have a dedicated node for > testing > as a standalone cluster? Or your node is part of the rest of the cluster > and you use it as a testing machine. > > In the first case the instrumentation is fine. In the second case it will > not work > since we modified the plugin and not a daemon only. We cannot have a node in > the cluster running with different plugins. Well, that's the second case: I can dedicate a test node, which is part of our production cluster, to test this. But it's just running slurmd, not slurmctld. So I guess if I run the instrumented plugin cluster-wide, our users will experience the segfaults again, right?
If you run the instrumented plugin cluster wide, which means install and restart the whole cluster, your users will not experienced core, at least not this one :-), because beside instrumentation I also check for the error case and handle it. I think that having an isolated host/virtual machine would be better for testing. For example I use vms quite a bit. David
(In reply to David Bigagli from comment #15) > If you run the instrumented plugin cluster wide, which means install and > restart the whole cluster, your users will not experienced core, at least > not this one :-), because beside instrumentation I also check for the error > case and handle it. > > I think that having an isolated host/virtual machine would be better for > testing. > For example I use vms quite a bit. But then running it on a VM probably wouldn't provide more information than what you already collected on your end, right? I guess I'm confused about the purpose of me running the instrumented plugin: is it to gather more debug information, or are you suggesting to run it as a workaround?
Hi, sorry if I confused you. I wanted to see if we would observe the same behavior on your cluster, verify that we see the same output from the instrumentation so that we can conclude it is really the same issue. If mkdir() succeed if we slow the code down a bit by printing some info then perhaps we can include it in the patch. It is important that mkdir() of the last leaf happens otherwise you will be missing memory accounting data. Ont he other hand the accounting using cgroups does not give you much, just cpu and memory, the rest comes from /proc anyway. I think that you could just use jobacct_gather/linux and be done with it. I will fix the bug making sure the slurmstepd does not core dump. Then you don't have to run the instrumentation. Many options :-) I am ok with any just let me know. I still have to fix the bug :-) David
(In reply to David Bigagli from comment #17) > Hi, sorry if I confused you. I wanted to see if we would observe the same > behavior on your cluster, verify that we see the same output from the > instrumentation so that we can conclude it is really the same issue. Ok. > If mkdir() succeed if we slow the code down a bit by printing some info then > perhaps we can include it in the patch. It is important that mkdir() of the > last leaf happens otherwise you will be missing memory accounting data. > Ont he other hand the accounting using cgroups does not give you much, just > cpu and memory, the rest comes from /proc anyway. The docs (http://slurm.schedmd.com/cgroups.html) seem to imply that the blkio subsystem is used too, and that's what prompted be to try it. > I think that you could just use jobacct_gather/linux and be done with it. > I will fix the bug making sure the slurmstepd does not core dump. Ok, I think that's good, and that's the primary reason I reported this (fixing the segfault). I don't absolutely *need* cgroups accounting, just wanted to try it. I thought that maybe it would also be more accurate than the parsing of /proc, in the sense that we're using proctrack/cgroup to set the limits, so information would come from the same source. > Then you don't have to run the instrumentation. Many options :-) > I am ok with any just let me know. I still have to fix the bug :-) Sure. :) I'll be happy with jobacct_gather/linux for now. But thanks for looking into it anyway!
Commit 3cd5b230. Most likely the problem will go away in newer kernels. David
(In reply to David Bigagli from comment #19) > Commit 3cd5b230. Most likely the problem will go away in newer kernels. Excellent, thank you very much.
