SchedMD - Slurm Support – Full Text Ticket Listing
- Home
- | New
- | Browse
- | Search
- | [?]
- | Reports
- | Help
- | New Account
- | Log In
- | Forgot Password
| Summary: | slurmd doesn't start in the training nodes in Scaleout | ||
|---|---|---|---|
| Product: | Slurm | Reporter: | Serguei Mokhov <serguei.mokhov> |
| Component: | Scaleout | Assignee: | Nate Rini <nrini> |
| Status: | RESOLVED TIMEDOUT | QA Contact: | |
| Severity: | 4 - Minor Issue | ||
| Priority: | --- | CC: | hpc, serguei.mokhov |
| Version: | 23.11.1 | ||
| Hardware: | Linux | ||
| OS: | Linux | ||
| Site: | Concordia University | Slinky Site: | --- |
| Alineos Sites: | --- | Atos/Eviden Sites: | --- |
| Confidential Site: | --- | Coreweave sites: | --- |
| Cray Sites: | --- | DS9 clusters: | --- |
| Google sites: | --- | HPCnow Sites: | --- |
| HPE Sites: | --- | IBM Sites: | --- |
| NOAA SIte: | --- | NoveTech Sites: | --- |
| Nvidia HWinf-CS Sites: | --- | OCF Sites: | --- |
| Recursion Pharma Sites: | --- | SFW Sites: | --- |
| SNIC sites: | --- | Tzag Elita Sites: | --- |
| Linux Distro: | AlmaLinux | Machine Name: | |
| CLE Version: | Version Fixed: | ||
| Target Release: | --- | DevPrio: | --- |
| Emory-Cloud Sites: | --- | ||
|
Description
Serguei Mokhov
2024-01-12 11:18:36 MST
(In reply to Serguei Mokhov from comment #0) > This is a fork of a ticket from bug ID 18620 for this issue: Thanks. Having 1 issue per ticket really helps us keep track of all of the issues and avoid a lot of confusion. > - It seems slurmd failed to start on each node[00-09] The relevant log was in comment#0: > Jan 12 04:14:29 node07 slurmd[121]: slurmd: fatal: Hybrid mode is not supported. Mounted cgroups are: 9:devices:/ As noted here: > https://slurm.schedmd.com/cgroups.html The hybrid cgroup mode is not supported by Slurm and has been removed in more recent kernels. Please configure cgroups v1 or v2 per the above link on the host system to resolve this error. > - it appears the cluster added slurm_pam_adopt but ssh still allows through One of the training labs includes adding the pam_deny to /etc/pam.d/sshd2 to enforce pam_slurm_adopt. This will need to be done manually if you wish to see it enforced. > - I see idle cloud[xxx] nodes?? I did not do a cloud build they are included in the default config to avoid having multiple versions. if not built with cloud mode, then they can be removed from slurm.conf or safely ignored. > - I can't ssh to the 'db' note and check it but can ping it Try calling `make HOST=db bash` instead. The container image is provided the the mysql project and does not include sshd. (In reply to Nate Rini from comment #1) > > - It seems slurmd failed to start on each node[00-09] > > The relevant log was in comment#0: > > Jan 12 04:14:29 node07 slurmd[121]: slurmd: fatal: Hybrid mode is not supported. Mounted cgroups are: 9:devices:/ > > As noted here: > > https://slurm.schedmd.com/cgroups.html > > The hybrid cgroup mode is not supported by Slurm and has been removed in > more recent kernels. Please configure cgroups v1 or v2 per the above link on > the host system to resolve this error. Oh, that's the parent system's cgroups not in the containers... Let me see how I can fix that... > > - it appears the cluster added slurm_pam_adopt but ssh still allows through > > One of the training labs includes adding the pam_deny to /etc/pam.d/sshd2 to > enforce pam_slurm_adopt. This will need to be done manually if you wish to > see it enforced. > > > - I see idle cloud[xxx] nodes?? I did not do a cloud build > > they are included in the default config to avoid having multiple versions. > if not built with cloud mode, then they can be removed from slurm.conf or > safely ignored. > > > - I can't ssh to the 'db' note and check it but can ping it > > Try calling `make HOST=db bash` instead. The container image is provided the > the mysql project and does not include sshd. Alright, thanks. (In reply to Serguei Mokhov from comment #2) > (In reply to Nate Rini from comment #1) > > > > - It seems slurmd failed to start on each node[00-09] > > > > The relevant log was in comment#0: > > > Jan 12 04:14:29 node07 slurmd[121]: slurmd: fatal: Hybrid mode is not supported. Mounted cgroups are: 9:devices:/ > > > > As noted here: > > > https://slurm.schedmd.com/cgroups.html > > > > The hybrid cgroup mode is not supported by Slurm and has been removed in > > more recent kernels. Please configure cgroups v1 or v2 per the above link on > > the host system to resolve this error. > > Oh, that's the parent system's cgroups not in the containers... > Let me see how I can fix that... I don't think the host system has hybrid or v1 cgroups: # cat /proc/cgroups #subsys_name hierarchy num_cgroups enabled cpuset 0 837 1 cpu 0 837 1 cpuacct 0 837 1 blkio 0 837 1 memory 0 837 1 devices 9 53 1 freezer 7 1 1 net_cls 5 1 1 perf_event 6 1 1 net_prio 5 1 1 hugetlb 0 837 1 pids 0 837 1 rdma 0 837 1 misc 0 837 1 # ll /sys/fs/cgroup/ total 0 -r--r--r-- 1 root root 0 Dec 18 19:22 cgroup.controllers -rw-r--r-- 1 root root 0 Jan 12 14:06 cgroup.max.depth -rw-r--r-- 1 root root 0 Jan 12 14:06 cgroup.max.descendants -rw-r--r-- 1 root root 0 Dec 18 19:22 cgroup.procs -r--r--r-- 1 root root 0 Jan 12 14:06 cgroup.stat -rw-r--r-- 1 root root 0 Jan 12 10:25 cgroup.subtree_control -rw-r--r-- 1 root root 0 Jan 12 14:06 cgroup.threads -r--r--r-- 1 root root 0 Jan 8 23:41 cpuset.cpus.effective -r--r--r-- 1 root root 0 Jan 8 23:41 cpuset.mems.effective -r--r--r-- 1 root root 0 Jan 12 14:06 cpu.stat drwxr-xr-x 2 root root 0 Jan 8 23:47 dev-hugepages.mount drwxr-xr-x 2 root root 0 Jan 8 23:47 dev-mqueue.mount drwxr-xr-x 24 root root 0 Jan 12 10:08 docker.slice drwxr-xr-x 2 root root 0 Dec 18 19:22 init.scope -r--r--r-- 1 root root 0 Jan 12 14:06 io.stat drwxr-xr-x 2 root root 0 Jan 8 23:47 machine.slice -r--r--r-- 1 root root 0 Jan 12 14:06 memory.numa_stat --w------- 1 root root 0 Jan 12 14:06 memory.reclaim -r--r--r-- 1 root root 0 Jan 12 14:06 memory.stat -r--r--r-- 1 root root 0 Jan 12 14:06 misc.capacity drwxr-xr-x 2 root root 0 Jan 8 23:47 proc-sys-fs-binfmt_misc.mount drwxr-xr-x 2 root root 0 Jan 8 23:47 sys-fs-fuse-connections.mount drwxr-xr-x 2 root root 0 Jan 8 23:47 sys-kernel-config.mount drwxr-xr-x 2 root root 0 Jan 8 23:47 sys-kernel-debug.mount drwxr-xr-x 2 root root 0 Jan 8 23:47 sys-kernel-tracing.mount drwxr-xr-x 48 root root 0 Jan 12 13:52 system.slice drwxr-xr-x 3 root root 0 Jan 12 11:44 user.slice [root@filth docker-scale-out]# cat /sys/fs/cgroup/cgroup.controllers cpuset cpu io memory hugetlb pids rdma misc [root@filth docker-scale-out]# stat -f /sys/fs/cgroup File: "/sys/fs/cgroup" ID: 0 Namelen: 255 Type: cgroup2fs Block size: 4096 Fundamental block size: 4096 Blocks: Total: 0 Free: 0 Available: 0 Inodes: Total: 0 Free: 0 How does it detect "hybrid"? (In reply to Serguei Mokhov from comment #3) > How does it detect "hybrid"? Call: > cat /proc/self/cgroup (In reply to Nate Rini from comment #4) > (In reply to Serguei Mokhov from comment #3) > > How does it detect "hybrid"? > > Call: > > cat /proc/self/cgroup # cat /proc/self/cgroup 9:devices:/ 7:freezer:/ 6:perf_event:/ 5:net_cls,net_prio:/ 1:name=systemd:/ 0::/user.slice/user-11929.slice/session-377.scope What here does tell me it is hybrid? (In reply to Serguei Mokhov from comment #5) > (In reply to Nate Rini from comment #4) > > (In reply to Serguei Mokhov from comment #3) > > > How does it detect "hybrid"? > > > > Call: > > > cat /proc/self/cgroup > > # cat /proc/self/cgroup > 9:devices:/ > 7:freezer:/ > 6:perf_event:/ > 5:net_cls,net_prio:/ > 1:name=systemd:/ > 0::/user.slice/user-11929.slice/session-377.scope > > What here does tell me it is hybrid? Cgroup v2 looks like this: > srun cat /proc/self/cgroup > 0::/system.slice/slurmstepd.scope/job_1330/step_0/user/task_0 Please try following the suggestions below: (In reply to Ben Glines from bug#18359 comment#8) > We only support legacy mode (cgroup v1) and unified mode (cgroup v2), and > not hybrid setups [3] as you have noticed. > > To disable hybrid mode and only enable cgroup v2 (this is the mode what we > recommend), you'll need to add "cgroup_no_v1=all" to your kernel command > line. Depending on your setup, this can be added to GRUB_CMDLINE_LINUX="" in > /etc/default/grub. Run `sudo update-grab` after making the change, and then > reboot. > > Check the cgroup mount points to ensure that onlyu cgroup v2 is enabled. You > should see something like the following: > > $ mount | grep cgroup > > cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot) > > Let me know if you have any questions about this. > > [1] https://slurm.schedmd.com/slurm.conf.html#OPT_task/affinity > [2] https://slurm.schedmd.com/cgroups.html#task > [3] https://slurm.schedmd.com/cgroups.html#overview (In reply to Nate Rini from comment #6) > (In reply to Serguei Mokhov from comment #5) > > (In reply to Nate Rini from comment #4) > > > (In reply to Serguei Mokhov from comment #3) > > > > How does it detect "hybrid"? > > > > > > Call: > > > > cat /proc/self/cgroup > > > > # cat /proc/self/cgroup > > 9:devices:/ > > 7:freezer:/ > > 6:perf_event:/ > > 5:net_cls,net_prio:/ > > 1:name=systemd:/ > > 0::/user.slice/user-11929.slice/session-377.scope > > > > What here does tell me it is hybrid? > > Cgroup v2 looks like this: > > srun cat /proc/self/cgroup > > 0::/system.slice/slurmstepd.scope/job_1330/step_0/user/task_0 Well I have 0::/ in mine too :) > Please try following the suggestions below: That's the thing... # mount | grep cgroup cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime) Looks I have nothing else mounted # uname -a Linux filth 5.14.0-362.8.1.el9_3.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Nov 7 14:54:22 EST 2023 x86_64 x86_64 x86_64 GNU/Linux "RHEL 9, by default, mounts and uses cgroups-v2." We did not change that in Alma. And AlmaLinux 9 defaults to using cgroup v2 as well. I'll try GRUB_CMDLINE_LINUX maybe next time I have the opportunity to reboot the server, but the above analysis so far suggests I don't have v1.. > (In reply to Ben Glines from bug#18359 comment#8) > > We only support legacy mode (cgroup v1) and unified mode (cgroup v2), and > > not hybrid setups [3] as you have noticed. > > > > To disable hybrid mode and only enable cgroup v2 (this is the mode what we > > recommend), you'll need to add "cgroup_no_v1=all" to your kernel command > > line. Depending on your setup, this can be added to GRUB_CMDLINE_LINUX="" in > > /etc/default/grub. Run `sudo update-grab` after making the change, and then > > reboot. > > > > Check the cgroup mount points to ensure that onlyu cgroup v2 is enabled. You > > should see something like the following: > > > $ mount | grep cgroup > > > cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot) > > > > Let me know if you have any questions about this. > > > > [1] https://slurm.schedmd.com/slurm.conf.html#OPT_task/affinity > > [2] https://slurm.schedmd.com/cgroups.html#task > > [3] https://slurm.schedmd.com/cgroups.html#overview So... I've added cgroup_no_v1=all to my GRUB_CMDLINE_LINUX and rebooted.
The result is virtually the same:
# cat /proc/self/cgroup
14:freezer:/
9:perf_event:/
8:net_cls,net_prio:/
7:devices:/
1:name=systemd:/
0::/user.slice/user-11929.slice/session-1.scope
# mount | grep cgroup
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
# make HOST=node01 bash
docker compose exec node01 /bin/bash
[root@node01 /]# systemctl status slurmd
● slurmd.service - Slurm node daemon
Loaded: loaded (/usr/lib/systemd/system/slurmd.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/slurmd.service.d
└─cluster.conf
/etc/systemd/system/slurmd.service.d
└─local.conf
Active: failed (Result: exit-code) since Fri 2024-01-12 19:10:20 EST; 6min ago
Process: 121 ExecStart=/usr/local/sbin/slurmd --systemd $SLURMD_OPTIONS (code=exited, status=1/FAILURE)
Process: 114 ExecStartPre=/usr/local/bin/slurmd.startup.sh (code=exited, status=0/SUCCESS)
Process: 113 ExecStartPre=/usr/bin/chown slurm:slurm /var/log/slurmd.log (code=exited, status=0/SUCCESS)
Process: 112 ExecStartPre=/usr/bin/touch /var/log/slurmd.log (code=exited, status=0/SUCCESS)
Main PID: 121 (code=exited, status=1/FAILURE)
Jan 12 19:10:20 node01 slurmd[121]: slurmd: error: ProctrackType 1 specified more than once, latest value used
Jan 12 19:10:20 node01 slurmd[121]: slurmd: debug: Log file re-opened
Jan 12 19:10:20 node01 slurmd[121]: slurmd: debug: CPUs:48 Boards:1 Sockets:2 CoresPerSocket:12 ThreadsPerCore:2
Jan 12 19:10:20 node01 slurmd[121]: slurmd: fatal: Hybrid mode is not supported. Mounted cgroups are: 14:freezer:/
Jan 12 19:10:20 node01 slurmd[121]: 9:perf_event:/
Jan 12 19:10:20 node01 slurmd[121]: 8:net_cls,net_prio:/
Jan 12 19:10:20 node01 slurmd[121]: 7:devices:/
Jan 12 19:10:20 node01 slurmd[121]: 1:name=systemd:/
Jan 12 19:10:20 node01 slurmd[121]: 0::/docker.slice/docker-4b68f66a75745866e007e7e21c61bac490d250d59925f6eb30246abf2dc22e3c.scope/init.scope
[root@node01 /]# cat /proc/self/cgroup
14:freezer:/
9:perf_event:/
8:net_cls,net_prio:/
7:devices:/
1:name=systemd:/
0::/docker.slice/docker-4b68f66a75745866e007e7e21c61bac490d250d59925f6eb30246abf2dc22e3c.scope/init.scope
[root@node01 /]# mount | grep cgroup
cgroup2 on /sys/fs/cgroup type cgroup2 (ro,relatime)
cgroup2 on /sys/fs/cgroup/docker.slice type cgroup2 (rw,nosuid,nodev,noexec,relatime)
[root@node01 /]#
(In reply to Serguei Mokhov from comment #8) > So... I've added cgroup_no_v1=all to my GRUB_CMDLINE_LINUX and rebooted. > The result is virtually the same: Is this after a reboot? (In reply to Nate Rini from comment #9) > (In reply to Serguei Mokhov from comment #8) > > So... I've added cgroup_no_v1=all to my GRUB_CMDLINE_LINUX and rebooted. > > The result is virtually the same: > > Is this after a reboot? Please try the suggestions here too: https://slurm.schedmd.com/faq.html#cgroupv2 The output in comment#7 is clearly a hybrid cgroup mount setup. Any change to try the suggestion in comment#10? It's been more than a week since comment#10. Please respond when convenient, and the ticket will automatically re-open. |