Ticket 18620

Summary: scaleout: slurm build fails
Product: Slurm Reporter: Nate Rini <nrini>
Component: OtherAssignee: Nate Rini <nrini>
Status: RESOLVED FIXED QA Contact:
Severity: 4 - Minor Issue    
Priority: --- CC: agilmoreno, hpc, serguei.mokhov
Version: 24.05.x   
Hardware: Linux   
OS: Linux   
Site: Concordia University Slinky Site: ---
Alineos Sites: --- Atos/Eviden Sites: ---
Confidential Site: --- Coreweave sites: ---
Cray Sites: --- DS9 clusters: ---
Google sites: --- HPCnow Sites: ---
HPE Sites: --- IBM Sites: ---
NOAA SIte: --- NoveTech Sites: ---
Nvidia HWinf-CS Sites: --- OCF Sites: ---
Recursion Pharma Sites: --- SFW Sites: ---
SNIC sites: --- Tzag Elita Sites: ---
Linux Distro: --- Machine Name:
CLE Version: Version Fixed: 24.08.0rc1
Target Release: --- DevPrio: ---
Emory-Cloud Sites: ---

Description Nate Rini 2024-01-09 09:17:41 MST 
99 [slurmdbd base  37/119] RUN git config --global user.email "you@example.com" &&     git config --global user.name "scale out" &&    git fetch --depth 1 origin master &&    git checkout FETCH_HEAD
#99 0.727 From https://github.com/SchedMD/slurm
#99 0.727  * branch            master     -> FETCH_HEAD
#99 0.834 Note: switching to 'FETCH_HEAD'.
#99 0.834 
#99 0.834 You are in 'detached HEAD' state. You can look around, make experimental
#99 0.834 changes and commit them, and you can discard any commits you make in this
#99 0.834 state without impacting any branches by switching back to a branch.
#99 0.834 
#99 0.834 If you want to create a new branch to retain commits you create, you may
#99 0.834 do so (now or later) by using -c with the switch command. Example:
#99 0.834 
#99 0.834   git switch -c <new-branch-name>
#99 0.834 
#99 0.834 Or undo this operation with:
#99 0.834 
#99 0.834   git switch -
#99 0.834 
#99 0.834 Turn off this advice by setting config variable advice.detachedHead to false
#99 0.834 
#99 0.834 HEAD is now at ee041bd Merge remote-tracking branch 'origin/slurm-23.11'
#99 DONE 0.9s

#100 [slurmdbd base  38/119] RUN ./configure --enable-slurmrestd --enable-multiple-slurmd --enable-memory-leak-debug --enable-developer --disable-optimizations --with-munge=/usr/local/ --prefix=/usr/local    --sharedstatedir=/run/slurm/ --with-pmix=/usr/local       --enable-pam --sysconfdir=/etc/slurm --with-hdf5=/usr/local/bin/h5cc >/dev/null &&      make -j >/dev/null && make install >/dev/null && make -j install-contrib >/dev/null &&    cd contribs && make install &&  ln -nfs /usr/local/src/slurm/contribs/slurm_completion_help/slurm_completion.sh /etc/bash_completion.d/slurm_completion.sh
#100 4.629 configure: WARNING: unable to locate libnvidia-ml.so and/or nvml.h
#100 4.735 configure: WARNING: unable to locate librocm_smi64.so and/or rocm_smi.h
#100 4.756 configure: WARNING: unable to locate libze_loader.so and/or ze_api.h
#100 4.954 configure: WARNING: unable to locate librdkafka.so and/or rdkafka.h
#100 5.272 configure: WARNING: unable to locate DataWarp installation
#100 5.294 configure: WARNING: HPE Slingshot: unable to locate libcxi/libcxi.h
#100 5.434 configure: WARNING: unable to build man page html files without man2html
#100 13.76 data.c: In function 'data_check_match':
#100 13.78 data.c:2115:7: error: label at end of compound statement
#100 13.78   case TYPE_MAX:
#100 13.78        ^~~~~~~~
#100 13.89 make[5]: *** [Makefile:918: data.lo] Error 1
#100 13.89 make[5]: *** Waiting for unfinished jobs....
#100 15.70 make[4]: *** [Makefile:1287: ../../src/common/libcommon.la] Error 2
#100 15.70 make[3]: *** [Makefile:1287: ../../src/api/libslurmhelper.la] Error 2
#100 15.70 make[2]: *** [Makefile:520: all-recursive] Error 1
#100 15.70 make[1]: *** [Makefile:621: all-recursive] Error 1
#100 15.70 make: *** [Makefile:520: all] Error 2
#100 ERROR: process "/bin/sh -c ./configure --enable-slurmrestd --enable-multiple-slurmd --enable-memory-leak-debug --enable-developer --disable-optimizations --with-munge=/usr/local/ --prefix=/usr/local \t--sharedstatedir=/run/slurm/ --with-pmix=/usr/local \t--enable-pam --sysconfdir=/etc/slurm --with-hdf5=/usr/local/bin/h5cc >/dev/null && \tmake -j >/dev/null && make install >/dev/null && make -j install-contrib >/dev/null && \tcd contribs && make install && \tln -nfs /usr/local/src/slurm/contribs/slurm_completion_help/slurm_completion.sh /etc/bash_completion.d/slurm_completion.sh" did not complete successfully: exit code: 2
------
 > [slurmdbd base  38/119] RUN ./configure --enable-slurmrestd --enable-multiple-slurmd --enable-memory-leak-debug --enable-developer --disable-optimizations --with-munge=/usr/local/ --prefix=/usr/local      --sharedstatedir=/run/slurm/ --with-pmix=/usr/local       --enable-pam --sysconfdir=/etc/slurm --with-hdf5=/usr/local/bin/h5cc >/dev/null &&      make -j >/dev/null && make install >/dev/null && make -j install-contrib >/dev/null &&    cd contribs && make install &&  ln -nfs /usr/local/src/slurm/contribs/slurm_completion_help/slurm_completion.sh /etc/bash_completion.d/slurm_completion.sh:
13.78 data.c:2115:7: error: label at end of compound statement
13.78   case TYPE_MAX:
13.78        ^~~~~~~~
13.89 make[5]: *** [Makefile:918: data.lo] Error 1
13.89 make[5]: *** Waiting for unfinished jobs....
15.70 make[4]: *** [Makefile:1287: ../../src/common/libcommon.la] Error 2
15.70 make[3]: *** [Makefile:1287: ../../src/api/libslurmhelper.la] Error 2
15.70 make[2]: *** [Makefile:520: all-recursive] Error 1
15.70 make[1]: *** [Makefile:621: all-recursive] Error 1
15.70 make: *** [Makefile:520: all] Error 2
Comment 2 Serguei Mokhov 2024-01-09 09:31:44 MST 
Thank you. Sorry I should have openened this here in the first place :)
Comment 3 Nate Rini 2024-01-09 09:39:15 MST 
(In reply to Serguei Mokhov from comment #2)
> Thank you. Sorry I should have openened this here in the first place :)

I happened to see the other one posted. I have replicated the issue and it will be fixed shortly.
Comment 6 Serguei Mokhov 2024-01-10 08:35:33 MST 
Nade, I saw one of your merges, and following your suggestion
on GitLab, I did export SLURM_RELEASE=slurm-23.11. I also
pulled the fix and decided to clean first and ran into this:

# export SLURM_RELEASE=slurm-23.11
# make clean
unknown shorthand flag: 'c' in -compose
test -f ./docker-compose.yml && (docker compose kill -s SIGKILL; docker compose down --remove-orphans -t1 -v; unlink ./docker-compose.yml) || true
[+] Running 13/0kill
 ✔ Volume docker-scale-out_elastic_data01     Removed                                                                                                                                                         0.0s 
 ✔ Volume docker-scale-out_cluster-etc-slurm  Removed                                                                                                                                                         0.0s 
 ✔ Volume docker-scale-out_xdmod              Removed                                                                                                                                                         0.0s 
 ✔ Volume docker-scale-out_root-home          Removed                                                                                                                                                         0.0s 
 ✔ Volume docker-scale-out_slurmctld          Removed                                                                                                                                                         0.0s 
 ✔ Volume docker-scale-out_src                Removed                                                                                                                                                         0.0s 
 ✔ Volume docker-scale-out_elastic_data02     Removed                                                                                                                                                         0.0s 
 ✔ Volume docker-scale-out_elastic_data03     Removed                                                                                                                                                         0.0s 
 ✔ Volume docker-scale-out_container-shared   Removed                                                                                                                                                         0.0s 
 ✔ Volume docker-scale-out_home               Removed                                                                                                                                                         0.0s 
 ✔ Volume docker-scale-out_etc-ssh            Removed                                                                                                                                                         0.0s 
 ✔ Volume docker-scale-out_auth               Removed                                                                                                                                                         0.0s 
 ✔ Volume docker-scale-out_mail               Removed                                                                                                                                                         0.0s 
[ -f cloud_socket ] && unlink cloud_socket || true

The -c error seems to come from here:

./Makefile:IMAGES := $(shell $(DC) -compose config | awk '{if ($$1 == "image:") print $$2;}' | sort | uniq)

Would that not be just "$(DC) config"? That gave me no error, but this:

 make clean
no configuration file provided: not found
test -f ./docker-compose.yml && (docker compose kill -s SIGKILL; docker compose down --remove-orphans -t1 -v; unlink ./docker-compose.yml) || true
[ -f cloud_socket ] && unlink cloud_socket || true
Comment 7 Nate Rini 2024-01-10 10:01:22 MST 
Please provide the output of the following:
> docker compose version
> docker --version
Comment 8 Serguei Mokhov 2024-01-11 11:33:17 MST 
(In reply to Nate Rini from comment #7)
> Please provide the output of the following:
> > docker compose version
> > docker --version

$ docker compose version
Docker Compose version v2.21.0
$ docker --version
Docker version 24.0.7, build afdd53b
Comment 9 Nate Rini 2024-01-11 12:04:34 MST 
(In reply to Serguei Mokhov from comment #6)
> ./Makefile:IMAGES := $(shell $(DC) -compose config | awk '{if ($$1 ==
> "image:") print $$2;}' | sort | uniq)
> 
> Would that not be just "$(DC) config"? That gave me no error, but this:

That was correct. It has been fixed:
> https://github.com/naterini/docker-scale-out/commit/5feca432fdfb155b47503023fb042d56bb6a72f1
Comment 10 Serguei Mokhov 2024-01-11 14:10:12 MST 
(In reply to Nate Rini from comment #9)
> (In reply to Serguei Mokhov from comment #6)
> > ./Makefile:IMAGES := $(shell $(DC) -compose config | awk '{if ($$1 ==
> > "image:") print $$2;}' | sort | uniq)
> > 
> > Would that not be just "$(DC) config"? That gave me no error, but this:
> 
> That was correct. It has been fixed:
> > https://github.com/naterini/docker-scale-out/commit/5feca432fdfb155b47503023fb042d56bb6a72f1

Cool, thanks!

What's the difference between

  https://github.com/naterini/docker-scale-out

and

  https://gitlab.com/SchedMD/training/docker-scale-out/

? Which one is primary and which one is a mirror?
Comment 11 Nate Rini 2024-01-11 14:50:47 MST 
(In reply to Serguei Mokhov from comment #10)
> What's the difference between

None, they are mirrors.

>   https://github.com/naterini/docker-scale-out
> 
> and
> 
>   https://gitlab.com/SchedMD/training/docker-scale-out/
> 
> ? Which one is primary and which one is a mirror?

gitlab is the official and primary
Comment 12 Serguei Mokhov 2024-01-11 19:52:00 MST 
Thank you. I've forked your GH one for reference; in case.

I have applied both fixes and it's been building now for almost a day
by now. Is this normal?

I am now at:

#56 114720.1   Cleanup    : nss-pem-1.0.3-7.el7.x86_64                               166/198 
#56 114720.1   Cleanup    : nss-softokn-3.53.1-6.el7_9.x86_64                        167/198 
#56 116048.9   Cleanup    : libmount-2.23.2-65.el7.x86_64                            168/198 
#56 117368.6   Cleanup    : libblkid-2.23.2-65.el7.x86_64                            169/198 
#56 118671.5   Cleanup    : libssh2-1.8.0-4.el7.x86_64                               170/198 
#56 119995.4   Cleanup    : systemd-libs-219-78.el7_9.3.x86_64                       171/198 
#56 121290.0   Cleanup    : cyrus-sasl-lib-2.1.26-23.el7.x86_64                      172/198 
#56 123911.0   Cleanup    : coreutils-8.22-24.el7.x86_64                             173/198 
#56 123911.0   Cleanup    : krb5-libs-1.15.1-50.el7.x86_64                           174/198 
#56 125210.0   Cleanup    : 1:openssl-libs-1.0.2k-21.el7_9.x86_64                    175/198

and each "Cleanup" line takes several minutes to appear.

Running on:

    capabilities: smp vsyscall32
  *-core
       description: Motherboard
       physical id: 0
     *-memory
          description: System memory
          physical id: 6
          size: 130GiB
     *-cpu:0
          product: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz
          vendor: Intel Corp.
          physical id: 7
          bus info: cpu@0
          version: 6.62.4
          size: 2913MHz
          capacity: 3200MHz
          width: 64 bits
          capabilities: fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp x86-64 constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_fault epb pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts vnmi md_clear flush_l1d cpufreq
          configuration: microcode=1070
     *-cpu:1
          product: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz
          vendor: Intel Corp.
          physical id: 8
          bus info: cpu@1
          version: 6.62.4
          size: 3001MHz
          capacity: 3200MHz
          width: 64 bits
          capabilities: fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp x86-64 constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_fault epb pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts vnmi md_clear flush_l1d cpufreq
          configuration: microcode=1070

00:00.0 Host bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 DMI2 (rev 04)
00:01.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 1a (rev 04)
00:01.1 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 1b (rev 04)
00:02.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2a (rev 04)
00:02.1 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2b (rev 04)
00:02.2 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2c (rev 04)
00:02.3 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2d (rev 04)
00:03.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3a (rev 04)
00:03.1 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3b (rev 04)
00:03.2 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3c (rev 04)
00:03.3 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3d (rev 04)
00:04.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 0 (rev 04)
00:04.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 1 (rev 04)
00:04.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 2 (rev 04)
00:04.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 3 (rev 04)
00:04.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 4 (rev 04)
00:04.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 5 (rev 04)
00:04.6 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 6 (rev 04)
00:04.7 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 7 (rev 04)
00:05.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 VTd/Memory Map/Misc (rev 04)
00:05.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IIO RAS (rev 04)
00:05.4 PIC: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IOAPIC (rev 04)
00:11.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Virtual Root Port (rev 05)
00:1a.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #2 (rev 05)
00:1c.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 1 (rev b5)
00:1c.7 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 8 (rev b5)
00:1d.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #1 (rev 05)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a5)
00:1f.0 ISA bridge: Intel Corporation C600/X79 series chipset LPC Controller (rev 05)
01:00.0 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Slave Instrumentation & System Support (rev 05)
01:00.1 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200EH
01:00.2 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Management Processor Support and Messaging (rev 05)
01:00.4 USB controller: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller (rev 02)
03:00.0 RAID bus controller: Hewlett-Packard Company Smart Array Gen8 Controllers (rev 01)
04:00.0 Ethernet controller: Emulex Corporation OneConnect 10Gb NIC (be3) (rev 01)
04:00.1 Ethernet controller: Emulex Corporation OneConnect 10Gb NIC (be3) (rev 01)
1f:08.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 0 (rev 04)
1f:09.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 1 (rev 04)
1f:0a.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 0 (rev 04)
1f:0a.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 1 (rev 04)
1f:0a.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 2 (rev 04)
1f:0a.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 3 (rev 04)
1f:0b.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 UBOX Registers (rev 04)
1f:0b.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 UBOX Registers (rev 04)
1f:0c.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
1f:0c.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
1f:0c.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
1f:0c.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
1f:0c.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
1f:0c.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
1f:0d.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
1f:0d.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
1f:0d.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
1f:0d.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
1f:0d.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
1f:0d.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
1f:0e.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 0 (rev 04)
1f:0e.1 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 0 (rev 04)
1f:0f.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Target Address/Thermal Registers (rev 04)
1f:0f.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 RAS Registers (rev 04)
1f:0f.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
1f:0f.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
1f:0f.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
1f:0f.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
1f:10.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 0 (rev 04)
1f:10.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 1 (rev 04)
1f:10.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 0 (rev 04)
1f:10.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 1 (rev 04)
1f:10.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 2 (rev 04)
1f:10.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 3 (rev 04)
1f:10.6 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 2 (rev 04)
1f:10.7 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 3 (rev 04)
1f:13.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 R2PCIe (rev 04)
1f:13.1 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 R2PCIe (rev 04)
1f:13.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Registers (rev 04)
1f:13.5 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Performance Ring Monitoring (rev 04)
1f:16.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 System Address Decoder (rev 04)
1f:16.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Broadcast Registers (rev 04)
1f:16.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Broadcast Registers (rev 04)
1f:1c.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 1 (rev 04)
1f:1c.1 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 1 (rev 04)
1f:1d.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Target Address/Thermal Registers (rev 04)
1f:1d.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 RAS Registers (rev 04)
1f:1d.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder Registers (rev 04)
1f:1d.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder Registers (rev 04)
1f:1d.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder Registers (rev 04)
1f:1d.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder Registers (rev 04)
1f:1e.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 Thermal Control 0 (rev 04)
1f:1e.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 Thermal Control 1 (rev 04)
1f:1e.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 ERROR Registers 0 (rev 04)
1f:1e.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 ERROR Registers 1 (rev 04)
1f:1e.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 Thermal Control 2 (rev 04)
1f:1e.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 Thermal Control 3 (rev 04)
1f:1e.6 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 ERROR Registers 2 (rev 04)
1f:1e.7 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 ERROR Registers 3 (rev 04)
20:00.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port in DMI2 Mode (rev 04)
20:01.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 1a (rev 04)
20:01.1 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 1b (rev 04)
20:02.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2a (rev 04)
20:02.1 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2b (rev 04)
20:02.2 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2c (rev 04)
20:02.3 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2d (rev 04)
20:03.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3a (rev 04)
20:03.1 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3b (rev 04)
20:03.2 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3c (rev 04)
20:03.3 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3d (rev 04)
20:04.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 0 (rev 04)
20:04.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 1 (rev 04)
20:04.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 2 (rev 04)
20:04.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 3 (rev 04)
20:04.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 4 (rev 04)
20:04.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 5 (rev 04)
20:04.6 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 6 (rev 04)
20:04.7 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 7 (rev 04)
20:05.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 VTd/Memory Map/Misc (rev 04)
20:05.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IIO RAS (rev 04)
20:05.4 PIC: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IOAPIC (rev 04)
3f:08.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 0 (rev 04)
3f:09.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 1 (rev 04)
3f:0a.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 0 (rev 04)
3f:0a.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 1 (rev 04)
3f:0a.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 2 (rev 04)
3f:0a.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 3 (rev 04)
3f:0b.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 UBOX Registers (rev 04)
3f:0b.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 UBOX Registers (rev 04)
3f:0c.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
3f:0c.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
3f:0c.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
3f:0c.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
3f:0c.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
3f:0c.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
3f:0d.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
3f:0d.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
3f:0d.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
3f:0d.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
3f:0d.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
3f:0d.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
3f:0e.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 0 (rev 04)
3f:0e.1 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 0 (rev 04)
3f:0f.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Target Address/Thermal Registers (rev 04)
3f:0f.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 RAS Registers (rev 04)
3f:0f.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
3f:0f.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
3f:0f.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
3f:0f.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
3f:10.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 0 (rev 04)
3f:10.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 1 (rev 04)
3f:10.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 0 (rev 04)
3f:10.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 1 (rev 04)
3f:10.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 2 (rev 04)
3f:10.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 3 (rev 04)
3f:10.6 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 2 (rev 04)
3f:10.7 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 3 (rev 04)
3f:13.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 R2PCIe (rev 04)
3f:13.1 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 R2PCIe (rev 04)
3f:13.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Registers (rev 04)
3f:13.5 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Performance Ring Monitoring (rev 04)
3f:16.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 System Address Decoder (rev 04)
3f:16.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Broadcast Registers (rev 04)
3f:16.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Broadcast Registers (rev 04)
3f:1c.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 1 (rev 04)
3f:1c.1 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 1 (rev 04)
3f:1d.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Target Address/Thermal Registers (rev 04)
3f:1d.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 RAS Registers (rev 04)
3f:1d.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder Registers (rev 04)
3f:1d.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder Registers (rev 04)
3f:1d.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder Registers (rev 04)
3f:1d.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder Registers (rev 04)
3f:1e.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 Thermal Control 0 (rev 04)
3f:1e.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 Thermal Control 1 (rev 04)
3f:1e.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 ERROR Registers 0 (rev 04)
3f:1e.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 ERROR Registers 1 (rev 04)
3f:1e.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 Thermal Control 2 (rev 04)
3f:1e.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 Thermal Control 3 (rev 04)
3f:1e.6 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 ERROR Registers 2 (rev 04)
3f:1e.7 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 ERROR Registers 3 (rev 04)


root      770946  0.0  0.0 221312  2176 pts/2    S+   Jan10   0:00 make build
root      771151  0.0  0.0 2256100 37712 pts/2   Sl+  Jan10   0:06 docker compose up --build --remove-orphans -d
root      771172  0.2  0.0 3423916 70528 pts/2   Sl+  Jan10   4:57 /usr/libexec/docker/cli-plugins/docker-compose compose up --build --remove-orphans -d
root      771525  0.0  0.0 1385040 21316 ?       Sl   Jan10   0:00 runc --log /var/lib/docker/buildkit/executor/runc-log.json --log-format json run --bundle /var/lib/docker/buildkit/executor/kq1yqe018f1bdgot6f3a
root      771537  0.0  0.0 1433396 127400 ?      Ss   Jan10   0:38 /usr/bin/python /usr/bin/yum update --exclude google-chrome-* -y
root     1900381  100  0.0 1433396 106668 ?      R    21:44   1:51 /usr/bin/python /usr/bin/yum update --exclude google-chrome-* -y

$ free
               total        used        free      shared  buff/cache   available
Mem:       131420096     3285112      625884        4764   128570456   128134984

top - 21:51:00 up 24 days,  2:28,  2 users,  load average: 1.01, 1.03, 1.00
Tasks: 527 total,   2 running, 525 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.5 us,  1.6 sy,  0.0 ni, 97.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem : 128339.9 total,    640.2 free,   3211.3 used, 125524.9 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used. 125128.6 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                     
1900381 root      20   0 1433396 106668      0 R 100.0   0.1   6:13.99 yum                                                                                                                                         
4052523 root      20   0 2391608  57724  29464 S   1.0   0.0  41:25.09 containerd                                                                                                                                  
1901383 serguei   20   0  226396   4716   3460 R   0.7   0.0   0:00.14 top                                                                                                                                         
 771172 root      20   0 3423916  70528  37016 S   0.3   0.1   4:58.76 docker-compose                                                                                                                              
 830530 root      20   0       0      0      0 I   0.3   0.0   2:17.69 kworker/18:1-events                                                                                                                         
1889952 root      20   0       0      0      0 I   0.3   0.0   0:00.04 kworker/u130:1-writeback                                                                                                                    
1899478 serguei   20   0   16288   7356   5264 S   0.3   0.0   0:00.10 sshd
Comment 13 Serguei Mokhov 2024-01-12 08:04:41 MST 
#56 DONE 149818.2s
Comment 14 Nate Rini 2024-01-12 09:10:25 MST 
(In reply to Serguei Mokhov from comment #12)
> I have applied both fixes and it's been building now for almost a day
> by now. Is this normal?

Build times vary greatly based on CPU, RAM, and disk speed. It takes about an hour to build on my new dev workstation, but it took ~3hrs on my old one. Build time is generally a one time cost for users, I suggest using the save and load functions once the build is complete to avoid needing to do it again for your testing.
 
> and each "Cleanup" line takes several minutes to appear.

Docker's use of image layers can be very slow on systems without enough memory or slow disks. I suggest using something like iotop to see the effective speed of the operations.
 
> Running on:
>      *-memory
>           size: 130GiB
>      *-cpu:0
>           product: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz
The cores are quite slow, but that should be sufficient memory size. Having liquid cooling on the CPU tends to help a lot as the CPU will be able to go into turbo mode more often. For reference, my dev workstation is a 5.7GHz AMD Ryzen 9 7950X3D 16-Core Processor with liquid cooling. If your use case requires rebuilding a lot, it might just make sense to grab a faster AWS/GCP/etc. node for building for a few dollars and using the save function to bring the final image to your workstation for your testing.
Comment 15 Serguei Mokhov 2024-01-12 10:29:53 MST 
Thank you for your replies! The Training cluster is up, but
has a few issues.

[root@mgmtnode ~]# slurmd -V
slurm 23.11.1


- It seems slurmd failed to start on each node[00-09]
- it appears the cluster added slurm_pam_adopt but ssh still allows through
- I see idle cloud[xxx] nodes?? I did not do a cloud build
- I can't ssh to the 'db' note and check it but can ping it

I can freely otherwise ssh between the nodes, etc. Here are
some diagnostic excerpts:

[root@mgmtnode ~]# sinfo -la
Fri Jan 12 12:06:49 2024
PARTITION AVAIL  TIMELIMIT   JOB_SIZE ROOT OVERSUBS     GROUPS  NODES       STATE RESERVATION NODELIST
cloud        up   infinite 1-infinite   no       NO        all   1025       idle~             cloud[0000-1024]
debug*       up   infinite 1-infinite   no       NO        all     10    unknown*             node[00-09]

[root@mgmtnode ~]# ssh db
ssh: connect to host db port 22: Connection refused
[root@mgmtnode ~]# ping db
PING db(db (2001:db8:1:1::1:3)) 56 data bytes
64 bytes from db (2001:db8:1:1::1:3): icmp_seq=1 ttl=64 time=0.045 ms
64 bytes from db (2001:db8:1:1::1:3): icmp_seq=2 ttl=64 time=0.040 ms
64 bytes from db (2001:db8:1:1::1:3): icmp_seq=3 ttl=64 time=0.035 ms
64 bytes from db (2001:db8:1:1::1:3): icmp_seq=4 ttl=64 time=0.027 ms
^C
--- db ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3098ms
rtt min/avg/max/mdev = 0.027/0.036/0.045/0.009 ms

slurmctld is fine:

[root@mgmtnode ~]# systemctl status slurmctld
● slurmctld.service - Slurm controller daemon
   Loaded: loaded (/usr/lib/systemd/system/slurmctld.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/slurmctld.service.d
           └─cluster.conf
           /etc/systemd/system/slurmctld.service.d
           └─local.conf
   Active: active (running) since Fri 2024-01-12 04:15:03 EST; 7h ago
  Process: 173 ExecStartPost=/usr/local/bin/slurmctld.startup2.sh (code=exited, status=0/SUCCESS)
  Process: 113 ExecStartPre=/usr/local/bin/slurmctld.startup.sh (code=exited, status=0/SUCCESS)
  Process: 112 ExecStartPre=/usr/bin/chown slurm:slurm /var/log/slurmctld.log (code=exited, status=0/SUCCESS)
  Process: 111 ExecStartPre=/usr/bin/touch /var/log/slurmctld.log (code=exited, status=0/SUCCESS)
  Process: 110 ExecStartPre=/usr/bin/chmod -R 0770 /auth (code=exited, status=0/SUCCESS)
  Process: 107 ExecStartPre=/usr/bin/chown -R slurm:slurm /auth /etc/slurm/ (code=exited, status=0/SUCCESS)
 Main PID: 166 (slurmctld)
    Tasks: 36
   Memory: 46.2M
   CGroup: /docker.slice/docker-2b1c74d8bc42dee8dbe55d6a796b56bb3d4500586ca08fe0c94bd56f92196cc8.scope/system.slice/slurmctld.service
           ├─166 /usr/local/sbin/slurmctld --systemd
           └─167 slurmctld: slurmscriptd

Jan 12 11:52:19 mgmtnode slurmctld[166]: slurmctld: debug:  Spawning registration agent for node[00-09] 10 hosts
Jan 12 11:52:31 mgmtnode slurmctld[166]: slurmctld: debug:  Spawning registration agent for node[00-09] 10 hosts
Jan 12 11:52:31 mgmtnode slurmctld[166]: slurmctld: error: Nodes node[00-09] not responding
Jan 12 11:52:39 mgmtnode slurmctld[166]: slurmctld: debug:  Requesting control from backup controller mgmtnode2
Jan 12 11:52:39 mgmtnode slurmctld[166]: slurmctld: debug:  backup controller mgmtnode2 responding
Jan 12 11:52:43 mgmtnode slurmctld[166]: slurmctld: error: Nodes node[00-09] not responding
Jan 12 11:52:43 mgmtnode slurmctld[166]: slurmctld: debug:  Spawning registration agent for node[00-09] 10 hosts
Jan 12 11:52:52 mgmtnode slurmctld[166]: slurmctld: debug:  sched: Running job scheduler for full queue.
Jan 12 11:52:55 mgmtnode slurmctld[166]: slurmctld: debug:  Spawning registration agent for node[00-09] 10 hosts
Jan 12 11:52:55 mgmtnode slurmctld[166]: slurmctld: error: Nodes node[00-09] not responding

but slurmd even after a restart fails. They are all like that in each node I spot-checked:

[root@node07 ~]# systemctl list-units
  UNIT                                   LOAD   ACTIVE SUB       DESCRIPTION                                          
  -.mount                                loaded active mounted   Root Mount                                           
  dev-log.mount                          loaded active mounted   /dev/log                                             
  dev-mqueue.mount                       loaded active mounted   POSIX Message Queue File System                      
  etc-hostname.mount                     loaded active mounted   /etc/hostname                                        
  etc-hosts.mount                        loaded active mounted   /etc/hosts                                           
  etc-resolv.conf.mount                  loaded active mounted   /etc/resolv.conf                                     
  etc-slurm.mount                        loaded active mounted   /etc/slurm                                           
  etc-ssh.mount                          loaded active mounted   /etc/ssh                                             
  home.mount                             loaded active mounted   /home                                                
  proc-acpi.mount                        loaded active mounted   /proc/acpi                                           
  proc-bus.mount                         loaded active mounted   /proc/bus                                            
  proc-fs.mount                          loaded active mounted   /proc/fs                                             
  proc-irq.mount                         loaded active mounted   /proc/irq                                            
  proc-kcore.mount                       loaded active mounted   /proc/kcore                                          
  proc-keys.mount                        loaded active mounted   /proc/keys                                           
  proc-scsi.mount                        loaded active mounted   /proc/scsi                                           
  proc-sysrq\x2dtrigger.mount            loaded active mounted   /proc/sysrq-trigger                                  
  proc-timer_list.mount                  loaded active mounted   /proc/timer_list                                     
  root.mount                             loaded active mounted   root.mount                                           
  run-lock.mount                         loaded active mounted   /run/lock                                            
  srv-containers.mount                   loaded active mounted   /srv/containers                                      
● sys-fs-fuse-connections.mount          masked active mounted   sys-fs-fuse-connections.mount                        
  sys-fs-fuse.mount                      loaded active mounted   /sys/fs/fuse                                         
  tmp.mount                              loaded active mounted   Temporary Directory (/tmp)                           
  usr-local-src.mount                    loaded active mounted   /usr/local/src                                       
  usr-share-zoneinfo-UTC.mount           loaded active mounted   /usr/share/zoneinfo/UTC                              
  var-lib-journal.mount                  loaded active mounted   /var/lib/journal                                     
  var-spool-mail.mount                   loaded active mounted   /var/spool/mail                                      
  systemd-ask-password-console.path      loaded active waiting   Dispatch Password Requests to Console Directory Watch
  systemd-ask-password-wall.path         loaded active waiting   Forward Password Requests to Wall Directory Watch    
  init.scope                             loaded active running   System and Service Manager                           
  crond.service                          loaded active running   Command Scheduler                                    
  dbus.service                           loaded active running   D-Bus System Message Bus                             
  dracut-shutdown.service                loaded active exited    Restore /run/initramfs on shutdown                   
  ldconfig.service                       loaded active exited    Rebuild Dynamic Linker Cache                         
  munge.service                          loaded active running   MUNGE authentication service                         
  selinux-autorelabel-mark.service       loaded active exited    Mark the need to relabel after reboot                
● slurmd.service                         loaded failed failed    Slurm node daemon                                    
  sshd.service                           loaded active running   OpenSSH server daemon                                
  systemd-journal-catalog-update.service loaded active exited    Rebuild Journal Catalog                              
  systemd-journal-flush.service          loaded active exited    Flush Journal to Persistent Storage                  
  systemd-journald.service               loaded active running   Journal Service                                      
  systemd-sysusers.service               loaded active exited    Create System Users                                  
  systemd-tmpfiles-setup.service         loaded active exited    Create Volatile Files and Directories                
  systemd-update-done.service            loaded active exited    Update is Completed                                  
  systemd-update-utmp.service            loaded active exited    Update UTMP about System Boot/Shutdown               
  systemd-user-sessions.service          loaded active exited    Permit User Sessions                                 
  -.slice                                loaded active active    Root Slice                                           
[root@node07 ~]# systemctl slurmd status
Unknown operation slurmd.
[root@node07 ~]# systemctl status slurmd
● slurmd.service - Slurm node daemon
   Loaded: loaded (/usr/lib/systemd/system/slurmd.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/slurmd.service.d
           └─cluster.conf
           /etc/systemd/system/slurmd.service.d
           └─local.conf
   Active: failed (Result: exit-code) since Fri 2024-01-12 04:14:29 EST; 7h ago
  Process: 121 ExecStart=/usr/local/sbin/slurmd --systemd $SLURMD_OPTIONS (code=exited, status=1/FAILURE)
  Process: 114 ExecStartPre=/usr/local/bin/slurmd.startup.sh (code=exited, status=0/SUCCESS)
  Process: 113 ExecStartPre=/usr/bin/chown slurm:slurm /var/log/slurmd.log (code=exited, status=0/SUCCESS)
  Process: 112 ExecStartPre=/usr/bin/touch /var/log/slurmd.log (code=exited, status=0/SUCCESS)
 Main PID: 121 (code=exited, status=1/FAILURE)

Jan 12 04:14:29 node07 slurmd[121]: slurmd: error: ProctrackType 1 specified more than once, latest value used
Jan 12 04:14:29 node07 slurmd[121]: slurmd: debug:  Log file re-opened
Jan 12 04:14:29 node07 slurmd[121]: slurmd: debug:  CPUs:48 Boards:1 Sockets:2 CoresPerSocket:12 ThreadsPerCore:2
Jan 12 04:14:29 node07 slurmd[121]: slurmd: fatal: Hybrid mode is not supported. Mounted cgroups are: 9:devices:/
Jan 12 04:14:29 node07 slurmd[121]: 7:freezer:/
Jan 12 04:14:29 node07 slurmd[121]: 6:perf_event:/
Jan 12 04:14:29 node07 slurmd[121]: 5:net_cls,net_prio:/
Jan 12 04:14:29 node07 slurmd[121]: 1:name=systemd:/
Jan 12 04:14:29 node07 slurmd[121]: 0::/docker.slice/docker-07a88b0251dce9e294f193e08a3c5a8821a24e089f927876045b1542a43e1023.scope/init.scope
[root@node07 ~]# systemctl restart slurmd
Job for slurmd.service failed because the control process exited with error code.
See "systemctl status slurmd.service" and "journalctl -xe" for details.
[root@node07 ~]# journalctl -xe
-- The system journal process has started up, opened the journal
-- files for writing and is now ready to process requests.
Jan 12 04:14:29 node07 systemd-journald[70]: Runtime journal (/run/log/journal/040e07a4495d48b1925d320d1aa78b73) is 8.0M, max 4.0G, 3.9G free.
-- Subject: Disk space used by the journal
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
-- 
-- Runtime journal (/run/log/journal/040e07a4495d48b1925d320d1aa78b73) is currently using 8.0M.
-- Maximum allowed usage is set to 4.0G.
-- Leaving at least 4.0G free (of currently available 20.2G of disk space).
-- Enforced usage limit is thus 4.0G, of which 3.9G are still available.
-- 
-- The limits controlling how much disk space is used by the journal may
-- be configured with SystemMaxUse=, SystemKeepFree=, SystemMaxFileSize=,
-- RuntimeMaxUse=, RuntimeKeepFree=, RuntimeMaxFileSize= settings in
-- /etc/systemd/journald.conf. See journald.conf(5) for details.
Jan 12 04:14:29 node07 systemd-journald[70]: Runtime journal (/run/log/journal/040e07a4495d48b1925d320d1aa78b73) is 8.0M, max 4.0G, 3.9G free.
-- Subject: Disk space used by the journal
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
-- 
-- Runtime journal (/run/log/journal/040e07a4495d48b1925d320d1aa78b73) is currently using 8.0M.
-- Maximum allowed usage is set to 4.0G.
-- Leaving at least 4.0G free (of currently available 20.2G of disk space).
-- Enforced usage limit is thus 4.0G, of which 3.9G are still available.
-- 
-- The limits controlling how much disk space is used by the journal may
-- be configured with SystemMaxUse=, SystemKeepFree=, SystemMaxFileSize=,
-- RuntimeMaxUse=, RuntimeKeepFree=, RuntimeMaxFileSize= settings in
-- /etc/systemd/journald.conf. See journald.conf(5) for details.
Jan 12 04:14:29 node07 systemd-tmpfiles[76]: [/usr/local/lib/tmpfiles.d/munge.conf:1] Line references path below legacy directory /var/run/, updating /var/run/munge → /run/munge; please update the tmpfiles.d/ d>
Jan 12 04:14:29 node07 slurmd[121]: slurmd: error: ProctrackType 1 specified more than once, latest value used
Jan 12 04:14:29 node07 slurmd[121]: slurmd: debug:  Log file re-opened
Jan 12 04:14:29 node07 slurmd[121]: slurmd: debug:  CPUs:48 Boards:1 Sockets:2 CoresPerSocket:12 ThreadsPerCore:2
Jan 12 04:14:29 node07 slurmd[121]: slurmd: fatal: Hybrid mode is not supported. Mounted cgroups are: 9:devices:/
Jan 12 04:14:29 node07 slurmd[121]: 7:freezer:/
Jan 12 04:14:29 node07 slurmd[121]: 6:perf_event:/
Jan 12 04:14:29 node07 slurmd[121]: 5:net_cls,net_prio:/
Jan 12 04:14:29 node07 slurmd[121]: 1:name=systemd:/
Jan 12 04:14:29 node07 slurmd[121]: 0::/docker.slice/docker-07a88b0251dce9e294f193e08a3c5a8821a24e089f927876045b1542a43e1023.scope/init.scope
Jan 12 11:51:24 node07 slurmd[258]: slurmd: error: ProctrackType 1 specified more than once, latest value used
Jan 12 11:51:24 node07 slurmd[258]: slurmd: debug:  Log file re-opened
Jan 12 11:51:24 node07 slurmd[258]: slurmd: debug:  CPUs:48 Boards:1 Sockets:2 CoresPerSocket:12 ThreadsPerCore:2
Jan 12 11:51:24 node07 slurmd[258]: slurmd: fatal: Hybrid mode is not supported. Mounted cgroups are: 9:devices:/
Jan 12 11:51:24 node07 slurmd[258]: 7:freezer:/
Jan 12 11:51:24 node07 slurmd[258]: 6:perf_event:/
Jan 12 11:51:24 node07 slurmd[258]: 5:net_cls,net_prio:/
Jan 12 11:51:24 node07 slurmd[258]: 1:name=systemd:/
Jan 12 11:51:24 node07 slurmd[258]: 0::/docker.slice/docker-07a88b0251dce9e294f193e08a3c5a8821a24e089f927876045b1542a43e1023.scope/init.scope
 ESCOC


a78b73) is 8.0M, max 4.0G, 3.9G free.













a78b73) is 8.0M, max 4.0G, 3.9G free.













h below legacy directory /var/run/, updating /var/run/munge → /run/munge; please update the tmpfiles.d/ drop-in file accordingly.
 used

erCore:2
evices:/




876045b1542a43e1023.scope/init.scope
 used

erCore:2
evices:/




876045b1542a43e1023.scope/init.scope
 ESCOD
-- The system journal process has started up, opened the journal
-- files for writing and is now ready to process requests.
Jan 12 04:14:29 node07 systemd-journald[70]: Runtime journal (/run/log/journal/040e07a4495d48b1925d320d1aa78b73) is 8.0M, max 4.0G, 3.9G free.
-- Subject: Disk space used by the journal
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
-- 
-- Runtime journal (/run/log/journal/040e07a4495d48b1925d320d1aa78b73) is currently using 8.0M.
-- Maximum allowed usage is set to 4.0G.
-- Leaving at least 4.0G free (of currently available 20.2G of disk space).
-- Enforced usage limit is thus 4.0G, of which 3.9G are still available.
-- 
-- The limits controlling how much disk space is used by the journal may
-- be configured with SystemMaxUse=, SystemKeepFree=, SystemMaxFileSize=,
-- RuntimeMaxUse=, RuntimeKeepFree=, RuntimeMaxFileSize= settings in
-- /etc/systemd/journald.conf. See journald.conf(5) for details.
Jan 12 04:14:29 node07 systemd-journald[70]: Runtime journal (/run/log/journal/040e07a4495d48b1925d320d1aa78b73) is 8.0M, max 4.0G, 3.9G free.
-- Subject: Disk space used by the journal
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
-- 
-- Runtime journal (/run/log/journal/040e07a4495d48b1925d320d1aa78b73) is currently using 8.0M.
-- Maximum allowed usage is set to 4.0G.
-- Leaving at least 4.0G free (of currently available 20.2G of disk space).
-- Enforced usage limit is thus 4.0G, of which 3.9G are still available.
-- 
-- The limits controlling how much disk space is used by the journal may
-- be configured with SystemMaxUse=, SystemKeepFree=, SystemMaxFileSize=,
-- RuntimeMaxUse=, RuntimeKeepFree=, RuntimeMaxFileSize= settings in
-- /etc/systemd/journald.conf. See journald.conf(5) for details.
Jan 12 04:14:29 node07 systemd-tmpfiles[76]: [/usr/local/lib/tmpfiles.d/munge.conf:1] Line references path below legacy directory /var/run/, updating /var/run/munge → /run/munge; please update the tmpfiles.d/ d>
Jan 12 04:14:29 node07 slurmd[121]: slurmd: error: ProctrackType 1 specified more than once, latest value used
Jan 12 04:14:29 node07 slurmd[121]: slurmd: debug:  Log file re-opened
Jan 12 04:14:29 node07 slurmd[121]: slurmd: debug:  CPUs:48 Boards:1 Sockets:2 CoresPerSocket:12 ThreadsPerCore:2
Jan 12 04:14:29 node07 slurmd[121]: slurmd: fatal: Hybrid mode is not supported. Mounted cgroups are: 9:devices:/
Jan 12 04:14:29 node07 slurmd[121]: 7:freezer:/
Jan 12 04:14:29 node07 slurmd[121]: 6:perf_event:/
Jan 12 04:14:29 node07 slurmd[121]: 5:net_cls,net_prio:/
Jan 12 04:14:29 node07 slurmd[121]: 1:name=systemd:/
Jan 12 04:14:29 node07 slurmd[121]: 0::/docker.slice/docker-07a88b0251dce9e294f193e08a3c5a8821a24e089f927876045b1542a43e1023.scope/init.scope
Jan 12 11:51:24 node07 slurmd[258]: slurmd: error: ProctrackType 1 specified more than once, latest value used
Jan 12 11:51:24 node07 slurmd[258]: slurmd: debug:  Log file re-opened
Jan 12 11:51:24 node07 slurmd[258]: slurmd: debug:  CPUs:48 Boards:1 Sockets:2 CoresPerSocket:12 ThreadsPerCore:2
Jan 12 11:51:24 node07 slurmd[258]: slurmd: fatal: Hybrid mode is not supported. Mounted cgroups are: 9:devices:/
Jan 12 11:51:24 node07 slurmd[258]: 7:freezer:/
Jan 12 11:51:24 node07 slurmd[258]: 6:perf_event:/
Jan 12 11:51:24 node07 slurmd[258]: 5:net_cls,net_prio:/
Jan 12 11:51:24 node07 slurmd[258]: 1:name=systemd:/
Jan 12 11:51:24 node07 slurmd[258]: 0::/docker.slice/docker-07a88b0251dce9e294f193e08a3c5a8821a24e089f927876045b1542a43e1023.scope/init.scope

wilma's pam_slurm_adopt is not working (BTW, in our real cluster in el7 and el9 it does work, just reporting this with the test cluster I just built):

[root@node09 /]# uname -a
Linux node09 5.14.0-362.8.1.el9_3.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Nov 7 14:54:22 EST 2023 x86_64 x86_64 x86_64 GNU/Linux
[root@node09 /]# ssh wilma@node01
wilma@node01's password: 
Access denied by pam_slurm_adopt: you have no active jobs on this node
[wilma@node01 ~]$ netstat -pnat
(No info could be read for "-p": geteuid()=1013 but you should be root.)
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -                   
tcp        0      0 127.0.0.11:40891        0.0.0.0:*               LISTEN      -                   
tcp6       0      0 :::22                   :::*                    LISTEN      -                   
tcp6       0      0 2001:db8:1:1::5:11:22   2001:db8:1:1::5:1:58838 ESTABLISHED -                   
[wilma@node01 ~]$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         _gateway        0.0.0.0         UG    0      0        0 eth0
10.11.0.0       0.0.0.0         255.255.0.0     U     0      0        0 eth0


Main issue now for me to get slurmd working in node01-09. What could be the cause? Thanks!
Comment 16 Nate Rini 2024-01-12 10:51:28 MST 
(In reply to Serguei Mokhov from comment #15)
> - It seems slurmd failed to start on each node[00-09]
> - it appears the cluster added slurm_pam_adopt but ssh still allows through
> - I see idle cloud[xxx] nodes?? I did not do a cloud build
> - I can't ssh to the 'db' note and check it but can ping it

Please open 1 ticket per issue as this ticket has gone quite a bit beyond the original issue which is still waiting review to inclusion upstream.
Comment 17 Serguei Mokhov 2024-01-12 11:14:20 MST 
(In reply to Nate Rini from comment #16)
> (In reply to Serguei Mokhov from comment #15)
> > - It seems slurmd failed to start on each node[00-09]
> > - it appears the cluster added slurm_pam_adopt but ssh still allows through
> > - I see idle cloud[xxx] nodes?? I did not do a cloud build
> > - I can't ssh to the 'db' note and check it but can ping it
> 
> Please open 1 ticket per issue as this ticket has gone quite a bit beyond
> the original issue which is still waiting review to inclusion upstream.

Alright, will do, thank you! Sorry about that :)
Comment 19 Serguei Mokhov 2024-01-12 11:22:39 MST 
(In reply to Serguei Mokhov from comment #17)
> (In reply to Nate Rini from comment #16)
> > (In reply to Serguei Mokhov from comment #15)
> > > - It seems slurmd failed to start on each node[00-09]
> > > - it appears the cluster added slurm_pam_adopt but ssh still allows through
> > > - I see idle cloud[xxx] nodes?? I did not do a cloud build
> > > - I can't ssh to the 'db' note and check it but can ping it
> > 
> > Please open 1 ticket per issue as this ticket has gone quite a bit beyond
> > the original issue which is still waiting review to inclusion upstream.
> 
> Alright, will do, thank you! Sorry about that :)

Opened:

https://bugs.schedmd.com/show_bug.cgi?id=18664
Comment 20 Serguei Mokhov 2024-01-12 17:00:43 MST 
Nate, related to this bug I just ran into another in the Makefile
trying to make stop:

https://github.com/naterini/docker-scale-out/pull/1
Comment 25 Nate Rini 2024-01-16 17:29:57 MST 
This has been fixed upstream on the master branch.