99 [slurmdbd base 37/119] RUN git config --global user.email "you@example.com" && git config --global user.name "scale out" && git fetch --depth 1 origin master && git checkout FETCH_HEAD #99 0.727 From https://github.com/SchedMD/slurm #99 0.727 * branch master -> FETCH_HEAD #99 0.834 Note: switching to 'FETCH_HEAD'. #99 0.834 #99 0.834 You are in 'detached HEAD' state. You can look around, make experimental #99 0.834 changes and commit them, and you can discard any commits you make in this #99 0.834 state without impacting any branches by switching back to a branch. #99 0.834 #99 0.834 If you want to create a new branch to retain commits you create, you may #99 0.834 do so (now or later) by using -c with the switch command. Example: #99 0.834 #99 0.834 git switch -c <new-branch-name> #99 0.834 #99 0.834 Or undo this operation with: #99 0.834 #99 0.834 git switch - #99 0.834 #99 0.834 Turn off this advice by setting config variable advice.detachedHead to false #99 0.834 #99 0.834 HEAD is now at ee041bd Merge remote-tracking branch 'origin/slurm-23.11' #99 DONE 0.9s #100 [slurmdbd base 38/119] RUN ./configure --enable-slurmrestd --enable-multiple-slurmd --enable-memory-leak-debug --enable-developer --disable-optimizations --with-munge=/usr/local/ --prefix=/usr/local --sharedstatedir=/run/slurm/ --with-pmix=/usr/local --enable-pam --sysconfdir=/etc/slurm --with-hdf5=/usr/local/bin/h5cc >/dev/null && make -j >/dev/null && make install >/dev/null && make -j install-contrib >/dev/null && cd contribs && make install && ln -nfs /usr/local/src/slurm/contribs/slurm_completion_help/slurm_completion.sh /etc/bash_completion.d/slurm_completion.sh #100 4.629 configure: WARNING: unable to locate libnvidia-ml.so and/or nvml.h #100 4.735 configure: WARNING: unable to locate librocm_smi64.so and/or rocm_smi.h #100 4.756 configure: WARNING: unable to locate libze_loader.so and/or ze_api.h #100 4.954 configure: WARNING: unable to locate librdkafka.so and/or rdkafka.h #100 5.272 configure: WARNING: unable to locate DataWarp installation #100 5.294 configure: WARNING: HPE Slingshot: unable to locate libcxi/libcxi.h #100 5.434 configure: WARNING: unable to build man page html files without man2html #100 13.76 data.c: In function 'data_check_match': #100 13.78 data.c:2115:7: error: label at end of compound statement #100 13.78 case TYPE_MAX: #100 13.78 ^~~~~~~~ #100 13.89 make[5]: *** [Makefile:918: data.lo] Error 1 #100 13.89 make[5]: *** Waiting for unfinished jobs.... #100 15.70 make[4]: *** [Makefile:1287: ../../src/common/libcommon.la] Error 2 #100 15.70 make[3]: *** [Makefile:1287: ../../src/api/libslurmhelper.la] Error 2 #100 15.70 make[2]: *** [Makefile:520: all-recursive] Error 1 #100 15.70 make[1]: *** [Makefile:621: all-recursive] Error 1 #100 15.70 make: *** [Makefile:520: all] Error 2 #100 ERROR: process "/bin/sh -c ./configure --enable-slurmrestd --enable-multiple-slurmd --enable-memory-leak-debug --enable-developer --disable-optimizations --with-munge=/usr/local/ --prefix=/usr/local \t--sharedstatedir=/run/slurm/ --with-pmix=/usr/local \t--enable-pam --sysconfdir=/etc/slurm --with-hdf5=/usr/local/bin/h5cc >/dev/null && \tmake -j >/dev/null && make install >/dev/null && make -j install-contrib >/dev/null && \tcd contribs && make install && \tln -nfs /usr/local/src/slurm/contribs/slurm_completion_help/slurm_completion.sh /etc/bash_completion.d/slurm_completion.sh" did not complete successfully: exit code: 2 ------ > [slurmdbd base 38/119] RUN ./configure --enable-slurmrestd --enable-multiple-slurmd --enable-memory-leak-debug --enable-developer --disable-optimizations --with-munge=/usr/local/ --prefix=/usr/local --sharedstatedir=/run/slurm/ --with-pmix=/usr/local --enable-pam --sysconfdir=/etc/slurm --with-hdf5=/usr/local/bin/h5cc >/dev/null && make -j >/dev/null && make install >/dev/null && make -j install-contrib >/dev/null && cd contribs && make install && ln -nfs /usr/local/src/slurm/contribs/slurm_completion_help/slurm_completion.sh /etc/bash_completion.d/slurm_completion.sh: 13.78 data.c:2115:7: error: label at end of compound statement 13.78 case TYPE_MAX: 13.78 ^~~~~~~~ 13.89 make[5]: *** [Makefile:918: data.lo] Error 1 13.89 make[5]: *** Waiting for unfinished jobs.... 15.70 make[4]: *** [Makefile:1287: ../../src/common/libcommon.la] Error 2 15.70 make[3]: *** [Makefile:1287: ../../src/api/libslurmhelper.la] Error 2 15.70 make[2]: *** [Makefile:520: all-recursive] Error 1 15.70 make[1]: *** [Makefile:621: all-recursive] Error 1 15.70 make: *** [Makefile:520: all] Error 2
Thank you. Sorry I should have openened this here in the first place :)
(In reply to Serguei Mokhov from comment #2) > Thank you. Sorry I should have openened this here in the first place :) I happened to see the other one posted. I have replicated the issue and it will be fixed shortly.
Nade, I saw one of your merges, and following your suggestion on GitLab, I did export SLURM_RELEASE=slurm-23.11. I also pulled the fix and decided to clean first and ran into this: # export SLURM_RELEASE=slurm-23.11 # make clean unknown shorthand flag: 'c' in -compose test -f ./docker-compose.yml && (docker compose kill -s SIGKILL; docker compose down --remove-orphans -t1 -v; unlink ./docker-compose.yml) || true [+] Running 13/0kill ✔ Volume docker-scale-out_elastic_data01 Removed 0.0s ✔ Volume docker-scale-out_cluster-etc-slurm Removed 0.0s ✔ Volume docker-scale-out_xdmod Removed 0.0s ✔ Volume docker-scale-out_root-home Removed 0.0s ✔ Volume docker-scale-out_slurmctld Removed 0.0s ✔ Volume docker-scale-out_src Removed 0.0s ✔ Volume docker-scale-out_elastic_data02 Removed 0.0s ✔ Volume docker-scale-out_elastic_data03 Removed 0.0s ✔ Volume docker-scale-out_container-shared Removed 0.0s ✔ Volume docker-scale-out_home Removed 0.0s ✔ Volume docker-scale-out_etc-ssh Removed 0.0s ✔ Volume docker-scale-out_auth Removed 0.0s ✔ Volume docker-scale-out_mail Removed 0.0s [ -f cloud_socket ] && unlink cloud_socket || true The -c error seems to come from here: ./Makefile:IMAGES := $(shell $(DC) -compose config | awk '{if ($$1 == "image:") print $$2;}' | sort | uniq) Would that not be just "$(DC) config"? That gave me no error, but this: make clean no configuration file provided: not found test -f ./docker-compose.yml && (docker compose kill -s SIGKILL; docker compose down --remove-orphans -t1 -v; unlink ./docker-compose.yml) || true [ -f cloud_socket ] && unlink cloud_socket || true
Please provide the output of the following: > docker compose version > docker --version
(In reply to Nate Rini from comment #7) > Please provide the output of the following: > > docker compose version > > docker --version $ docker compose version Docker Compose version v2.21.0 $ docker --version Docker version 24.0.7, build afdd53b
(In reply to Serguei Mokhov from comment #6) > ./Makefile:IMAGES := $(shell $(DC) -compose config | awk '{if ($$1 == > "image:") print $$2;}' | sort | uniq) > > Would that not be just "$(DC) config"? That gave me no error, but this: That was correct. It has been fixed: > https://github.com/naterini/docker-scale-out/commit/5feca432fdfb155b47503023fb042d56bb6a72f1
(In reply to Nate Rini from comment #9) > (In reply to Serguei Mokhov from comment #6) > > ./Makefile:IMAGES := $(shell $(DC) -compose config | awk '{if ($$1 == > > "image:") print $$2;}' | sort | uniq) > > > > Would that not be just "$(DC) config"? That gave me no error, but this: > > That was correct. It has been fixed: > > https://github.com/naterini/docker-scale-out/commit/5feca432fdfb155b47503023fb042d56bb6a72f1 Cool, thanks! What's the difference between https://github.com/naterini/docker-scale-out and https://gitlab.com/SchedMD/training/docker-scale-out/ ? Which one is primary and which one is a mirror?
(In reply to Serguei Mokhov from comment #10) > What's the difference between None, they are mirrors. > https://github.com/naterini/docker-scale-out > > and > > https://gitlab.com/SchedMD/training/docker-scale-out/ > > ? Which one is primary and which one is a mirror? gitlab is the official and primary
Thank you. I've forked your GH one for reference; in case. I have applied both fixes and it's been building now for almost a day by now. Is this normal? I am now at: #56 114720.1 Cleanup : nss-pem-1.0.3-7.el7.x86_64 166/198 #56 114720.1 Cleanup : nss-softokn-3.53.1-6.el7_9.x86_64 167/198 #56 116048.9 Cleanup : libmount-2.23.2-65.el7.x86_64 168/198 #56 117368.6 Cleanup : libblkid-2.23.2-65.el7.x86_64 169/198 #56 118671.5 Cleanup : libssh2-1.8.0-4.el7.x86_64 170/198 #56 119995.4 Cleanup : systemd-libs-219-78.el7_9.3.x86_64 171/198 #56 121290.0 Cleanup : cyrus-sasl-lib-2.1.26-23.el7.x86_64 172/198 #56 123911.0 Cleanup : coreutils-8.22-24.el7.x86_64 173/198 #56 123911.0 Cleanup : krb5-libs-1.15.1-50.el7.x86_64 174/198 #56 125210.0 Cleanup : 1:openssl-libs-1.0.2k-21.el7_9.x86_64 175/198 and each "Cleanup" line takes several minutes to appear. Running on: capabilities: smp vsyscall32 *-core description: Motherboard physical id: 0 *-memory description: System memory physical id: 6 size: 130GiB *-cpu:0 product: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz vendor: Intel Corp. physical id: 7 bus info: cpu@0 version: 6.62.4 size: 2913MHz capacity: 3200MHz width: 64 bits capabilities: fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp x86-64 constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_fault epb pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts vnmi md_clear flush_l1d cpufreq configuration: microcode=1070 *-cpu:1 product: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz vendor: Intel Corp. physical id: 8 bus info: cpu@1 version: 6.62.4 size: 3001MHz capacity: 3200MHz width: 64 bits capabilities: fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp x86-64 constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_fault epb pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts vnmi md_clear flush_l1d cpufreq configuration: microcode=1070 00:00.0 Host bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 DMI2 (rev 04) 00:01.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 1a (rev 04) 00:01.1 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 1b (rev 04) 00:02.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2a (rev 04) 00:02.1 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2b (rev 04) 00:02.2 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2c (rev 04) 00:02.3 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2d (rev 04) 00:03.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3a (rev 04) 00:03.1 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3b (rev 04) 00:03.2 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3c (rev 04) 00:03.3 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3d (rev 04) 00:04.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 0 (rev 04) 00:04.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 1 (rev 04) 00:04.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 2 (rev 04) 00:04.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 3 (rev 04) 00:04.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 4 (rev 04) 00:04.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 5 (rev 04) 00:04.6 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 6 (rev 04) 00:04.7 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 7 (rev 04) 00:05.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 VTd/Memory Map/Misc (rev 04) 00:05.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IIO RAS (rev 04) 00:05.4 PIC: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IOAPIC (rev 04) 00:11.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Virtual Root Port (rev 05) 00:1a.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #2 (rev 05) 00:1c.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 1 (rev b5) 00:1c.7 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 8 (rev b5) 00:1d.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #1 (rev 05) 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a5) 00:1f.0 ISA bridge: Intel Corporation C600/X79 series chipset LPC Controller (rev 05) 01:00.0 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Slave Instrumentation & System Support (rev 05) 01:00.1 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200EH 01:00.2 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Management Processor Support and Messaging (rev 05) 01:00.4 USB controller: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller (rev 02) 03:00.0 RAID bus controller: Hewlett-Packard Company Smart Array Gen8 Controllers (rev 01) 04:00.0 Ethernet controller: Emulex Corporation OneConnect 10Gb NIC (be3) (rev 01) 04:00.1 Ethernet controller: Emulex Corporation OneConnect 10Gb NIC (be3) (rev 01) 1f:08.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 0 (rev 04) 1f:09.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 1 (rev 04) 1f:0a.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 0 (rev 04) 1f:0a.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 1 (rev 04) 1f:0a.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 2 (rev 04) 1f:0a.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 3 (rev 04) 1f:0b.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 UBOX Registers (rev 04) 1f:0b.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 UBOX Registers (rev 04) 1f:0c.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) 1f:0c.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) 1f:0c.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) 1f:0c.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) 1f:0c.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) 1f:0c.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) 1f:0d.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) 1f:0d.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) 1f:0d.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) 1f:0d.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) 1f:0d.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) 1f:0d.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) 1f:0e.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 0 (rev 04) 1f:0e.1 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 0 (rev 04) 1f:0f.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Target Address/Thermal Registers (rev 04) 1f:0f.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 RAS Registers (rev 04) 1f:0f.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04) 1f:0f.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04) 1f:0f.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04) 1f:0f.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04) 1f:10.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 0 (rev 04) 1f:10.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 1 (rev 04) 1f:10.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 0 (rev 04) 1f:10.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 1 (rev 04) 1f:10.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 2 (rev 04) 1f:10.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 3 (rev 04) 1f:10.6 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 2 (rev 04) 1f:10.7 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 3 (rev 04) 1f:13.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 R2PCIe (rev 04) 1f:13.1 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 R2PCIe (rev 04) 1f:13.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Registers (rev 04) 1f:13.5 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Performance Ring Monitoring (rev 04) 1f:16.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 System Address Decoder (rev 04) 1f:16.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Broadcast Registers (rev 04) 1f:16.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Broadcast Registers (rev 04) 1f:1c.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 1 (rev 04) 1f:1c.1 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 1 (rev 04) 1f:1d.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Target Address/Thermal Registers (rev 04) 1f:1d.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 RAS Registers (rev 04) 1f:1d.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder Registers (rev 04) 1f:1d.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder Registers (rev 04) 1f:1d.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder Registers (rev 04) 1f:1d.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder Registers (rev 04) 1f:1e.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 Thermal Control 0 (rev 04) 1f:1e.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 Thermal Control 1 (rev 04) 1f:1e.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 ERROR Registers 0 (rev 04) 1f:1e.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 ERROR Registers 1 (rev 04) 1f:1e.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 Thermal Control 2 (rev 04) 1f:1e.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 Thermal Control 3 (rev 04) 1f:1e.6 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 ERROR Registers 2 (rev 04) 1f:1e.7 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 ERROR Registers 3 (rev 04) 20:00.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port in DMI2 Mode (rev 04) 20:01.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 1a (rev 04) 20:01.1 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 1b (rev 04) 20:02.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2a (rev 04) 20:02.1 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2b (rev 04) 20:02.2 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2c (rev 04) 20:02.3 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2d (rev 04) 20:03.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3a (rev 04) 20:03.1 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3b (rev 04) 20:03.2 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3c (rev 04) 20:03.3 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3d (rev 04) 20:04.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 0 (rev 04) 20:04.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 1 (rev 04) 20:04.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 2 (rev 04) 20:04.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 3 (rev 04) 20:04.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 4 (rev 04) 20:04.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 5 (rev 04) 20:04.6 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 6 (rev 04) 20:04.7 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 7 (rev 04) 20:05.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 VTd/Memory Map/Misc (rev 04) 20:05.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IIO RAS (rev 04) 20:05.4 PIC: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IOAPIC (rev 04) 3f:08.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 0 (rev 04) 3f:09.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 1 (rev 04) 3f:0a.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 0 (rev 04) 3f:0a.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 1 (rev 04) 3f:0a.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 2 (rev 04) 3f:0a.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 3 (rev 04) 3f:0b.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 UBOX Registers (rev 04) 3f:0b.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 UBOX Registers (rev 04) 3f:0c.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) 3f:0c.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) 3f:0c.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) 3f:0c.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) 3f:0c.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) 3f:0c.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) 3f:0d.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) 3f:0d.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) 3f:0d.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) 3f:0d.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) 3f:0d.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) 3f:0d.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04) 3f:0e.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 0 (rev 04) 3f:0e.1 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 0 (rev 04) 3f:0f.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Target Address/Thermal Registers (rev 04) 3f:0f.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 RAS Registers (rev 04) 3f:0f.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04) 3f:0f.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04) 3f:0f.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04) 3f:0f.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04) 3f:10.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 0 (rev 04) 3f:10.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 1 (rev 04) 3f:10.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 0 (rev 04) 3f:10.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 1 (rev 04) 3f:10.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 2 (rev 04) 3f:10.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 3 (rev 04) 3f:10.6 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 2 (rev 04) 3f:10.7 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 3 (rev 04) 3f:13.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 R2PCIe (rev 04) 3f:13.1 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 R2PCIe (rev 04) 3f:13.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Registers (rev 04) 3f:13.5 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Performance Ring Monitoring (rev 04) 3f:16.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 System Address Decoder (rev 04) 3f:16.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Broadcast Registers (rev 04) 3f:16.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Broadcast Registers (rev 04) 3f:1c.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 1 (rev 04) 3f:1c.1 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 1 (rev 04) 3f:1d.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Target Address/Thermal Registers (rev 04) 3f:1d.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 RAS Registers (rev 04) 3f:1d.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder Registers (rev 04) 3f:1d.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder Registers (rev 04) 3f:1d.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder Registers (rev 04) 3f:1d.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder Registers (rev 04) 3f:1e.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 Thermal Control 0 (rev 04) 3f:1e.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 Thermal Control 1 (rev 04) 3f:1e.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 ERROR Registers 0 (rev 04) 3f:1e.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 ERROR Registers 1 (rev 04) 3f:1e.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 Thermal Control 2 (rev 04) 3f:1e.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 Thermal Control 3 (rev 04) 3f:1e.6 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 ERROR Registers 2 (rev 04) 3f:1e.7 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 ERROR Registers 3 (rev 04) root 770946 0.0 0.0 221312 2176 pts/2 S+ Jan10 0:00 make build root 771151 0.0 0.0 2256100 37712 pts/2 Sl+ Jan10 0:06 docker compose up --build --remove-orphans -d root 771172 0.2 0.0 3423916 70528 pts/2 Sl+ Jan10 4:57 /usr/libexec/docker/cli-plugins/docker-compose compose up --build --remove-orphans -d root 771525 0.0 0.0 1385040 21316 ? Sl Jan10 0:00 runc --log /var/lib/docker/buildkit/executor/runc-log.json --log-format json run --bundle /var/lib/docker/buildkit/executor/kq1yqe018f1bdgot6f3a root 771537 0.0 0.0 1433396 127400 ? Ss Jan10 0:38 /usr/bin/python /usr/bin/yum update --exclude google-chrome-* -y root 1900381 100 0.0 1433396 106668 ? R 21:44 1:51 /usr/bin/python /usr/bin/yum update --exclude google-chrome-* -y $ free total used free shared buff/cache available Mem: 131420096 3285112 625884 4764 128570456 128134984 top - 21:51:00 up 24 days, 2:28, 2 users, load average: 1.01, 1.03, 1.00 Tasks: 527 total, 2 running, 525 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.5 us, 1.6 sy, 0.0 ni, 97.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st MiB Mem : 128339.9 total, 640.2 free, 3211.3 used, 125524.9 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 125128.6 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1900381 root 20 0 1433396 106668 0 R 100.0 0.1 6:13.99 yum 4052523 root 20 0 2391608 57724 29464 S 1.0 0.0 41:25.09 containerd 1901383 serguei 20 0 226396 4716 3460 R 0.7 0.0 0:00.14 top 771172 root 20 0 3423916 70528 37016 S 0.3 0.1 4:58.76 docker-compose 830530 root 20 0 0 0 0 I 0.3 0.0 2:17.69 kworker/18:1-events 1889952 root 20 0 0 0 0 I 0.3 0.0 0:00.04 kworker/u130:1-writeback 1899478 serguei 20 0 16288 7356 5264 S 0.3 0.0 0:00.10 sshd
#56 DONE 149818.2s
(In reply to Serguei Mokhov from comment #12) > I have applied both fixes and it's been building now for almost a day > by now. Is this normal? Build times vary greatly based on CPU, RAM, and disk speed. It takes about an hour to build on my new dev workstation, but it took ~3hrs on my old one. Build time is generally a one time cost for users, I suggest using the save and load functions once the build is complete to avoid needing to do it again for your testing. > and each "Cleanup" line takes several minutes to appear. Docker's use of image layers can be very slow on systems without enough memory or slow disks. I suggest using something like iotop to see the effective speed of the operations. > Running on: > *-memory > size: 130GiB > *-cpu:0 > product: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz The cores are quite slow, but that should be sufficient memory size. Having liquid cooling on the CPU tends to help a lot as the CPU will be able to go into turbo mode more often. For reference, my dev workstation is a 5.7GHz AMD Ryzen 9 7950X3D 16-Core Processor with liquid cooling. If your use case requires rebuilding a lot, it might just make sense to grab a faster AWS/GCP/etc. node for building for a few dollars and using the save function to bring the final image to your workstation for your testing.
Thank you for your replies! The Training cluster is up, but has a few issues. [root@mgmtnode ~]# slurmd -V slurm 23.11.1 - It seems slurmd failed to start on each node[00-09] - it appears the cluster added slurm_pam_adopt but ssh still allows through - I see idle cloud[xxx] nodes?? I did not do a cloud build - I can't ssh to the 'db' note and check it but can ping it I can freely otherwise ssh between the nodes, etc. Here are some diagnostic excerpts: [root@mgmtnode ~]# sinfo -la Fri Jan 12 12:06:49 2024 PARTITION AVAIL TIMELIMIT JOB_SIZE ROOT OVERSUBS GROUPS NODES STATE RESERVATION NODELIST cloud up infinite 1-infinite no NO all 1025 idle~ cloud[0000-1024] debug* up infinite 1-infinite no NO all 10 unknown* node[00-09] [root@mgmtnode ~]# ssh db ssh: connect to host db port 22: Connection refused [root@mgmtnode ~]# ping db PING db(db (2001:db8:1:1::1:3)) 56 data bytes 64 bytes from db (2001:db8:1:1::1:3): icmp_seq=1 ttl=64 time=0.045 ms 64 bytes from db (2001:db8:1:1::1:3): icmp_seq=2 ttl=64 time=0.040 ms 64 bytes from db (2001:db8:1:1::1:3): icmp_seq=3 ttl=64 time=0.035 ms 64 bytes from db (2001:db8:1:1::1:3): icmp_seq=4 ttl=64 time=0.027 ms ^C --- db ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3098ms rtt min/avg/max/mdev = 0.027/0.036/0.045/0.009 ms slurmctld is fine: [root@mgmtnode ~]# systemctl status slurmctld ● slurmctld.service - Slurm controller daemon Loaded: loaded (/usr/lib/systemd/system/slurmctld.service; enabled; vendor preset: disabled) Drop-In: /usr/lib/systemd/system/slurmctld.service.d └─cluster.conf /etc/systemd/system/slurmctld.service.d └─local.conf Active: active (running) since Fri 2024-01-12 04:15:03 EST; 7h ago Process: 173 ExecStartPost=/usr/local/bin/slurmctld.startup2.sh (code=exited, status=0/SUCCESS) Process: 113 ExecStartPre=/usr/local/bin/slurmctld.startup.sh (code=exited, status=0/SUCCESS) Process: 112 ExecStartPre=/usr/bin/chown slurm:slurm /var/log/slurmctld.log (code=exited, status=0/SUCCESS) Process: 111 ExecStartPre=/usr/bin/touch /var/log/slurmctld.log (code=exited, status=0/SUCCESS) Process: 110 ExecStartPre=/usr/bin/chmod -R 0770 /auth (code=exited, status=0/SUCCESS) Process: 107 ExecStartPre=/usr/bin/chown -R slurm:slurm /auth /etc/slurm/ (code=exited, status=0/SUCCESS) Main PID: 166 (slurmctld) Tasks: 36 Memory: 46.2M CGroup: /docker.slice/docker-2b1c74d8bc42dee8dbe55d6a796b56bb3d4500586ca08fe0c94bd56f92196cc8.scope/system.slice/slurmctld.service ├─166 /usr/local/sbin/slurmctld --systemd └─167 slurmctld: slurmscriptd Jan 12 11:52:19 mgmtnode slurmctld[166]: slurmctld: debug: Spawning registration agent for node[00-09] 10 hosts Jan 12 11:52:31 mgmtnode slurmctld[166]: slurmctld: debug: Spawning registration agent for node[00-09] 10 hosts Jan 12 11:52:31 mgmtnode slurmctld[166]: slurmctld: error: Nodes node[00-09] not responding Jan 12 11:52:39 mgmtnode slurmctld[166]: slurmctld: debug: Requesting control from backup controller mgmtnode2 Jan 12 11:52:39 mgmtnode slurmctld[166]: slurmctld: debug: backup controller mgmtnode2 responding Jan 12 11:52:43 mgmtnode slurmctld[166]: slurmctld: error: Nodes node[00-09] not responding Jan 12 11:52:43 mgmtnode slurmctld[166]: slurmctld: debug: Spawning registration agent for node[00-09] 10 hosts Jan 12 11:52:52 mgmtnode slurmctld[166]: slurmctld: debug: sched: Running job scheduler for full queue. Jan 12 11:52:55 mgmtnode slurmctld[166]: slurmctld: debug: Spawning registration agent for node[00-09] 10 hosts Jan 12 11:52:55 mgmtnode slurmctld[166]: slurmctld: error: Nodes node[00-09] not responding but slurmd even after a restart fails. They are all like that in each node I spot-checked: [root@node07 ~]# systemctl list-units UNIT LOAD ACTIVE SUB DESCRIPTION -.mount loaded active mounted Root Mount dev-log.mount loaded active mounted /dev/log dev-mqueue.mount loaded active mounted POSIX Message Queue File System etc-hostname.mount loaded active mounted /etc/hostname etc-hosts.mount loaded active mounted /etc/hosts etc-resolv.conf.mount loaded active mounted /etc/resolv.conf etc-slurm.mount loaded active mounted /etc/slurm etc-ssh.mount loaded active mounted /etc/ssh home.mount loaded active mounted /home proc-acpi.mount loaded active mounted /proc/acpi proc-bus.mount loaded active mounted /proc/bus proc-fs.mount loaded active mounted /proc/fs proc-irq.mount loaded active mounted /proc/irq proc-kcore.mount loaded active mounted /proc/kcore proc-keys.mount loaded active mounted /proc/keys proc-scsi.mount loaded active mounted /proc/scsi proc-sysrq\x2dtrigger.mount loaded active mounted /proc/sysrq-trigger proc-timer_list.mount loaded active mounted /proc/timer_list root.mount loaded active mounted root.mount run-lock.mount loaded active mounted /run/lock srv-containers.mount loaded active mounted /srv/containers ● sys-fs-fuse-connections.mount masked active mounted sys-fs-fuse-connections.mount sys-fs-fuse.mount loaded active mounted /sys/fs/fuse tmp.mount loaded active mounted Temporary Directory (/tmp) usr-local-src.mount loaded active mounted /usr/local/src usr-share-zoneinfo-UTC.mount loaded active mounted /usr/share/zoneinfo/UTC var-lib-journal.mount loaded active mounted /var/lib/journal var-spool-mail.mount loaded active mounted /var/spool/mail systemd-ask-password-console.path loaded active waiting Dispatch Password Requests to Console Directory Watch systemd-ask-password-wall.path loaded active waiting Forward Password Requests to Wall Directory Watch init.scope loaded active running System and Service Manager crond.service loaded active running Command Scheduler dbus.service loaded active running D-Bus System Message Bus dracut-shutdown.service loaded active exited Restore /run/initramfs on shutdown ldconfig.service loaded active exited Rebuild Dynamic Linker Cache munge.service loaded active running MUNGE authentication service selinux-autorelabel-mark.service loaded active exited Mark the need to relabel after reboot ● slurmd.service loaded failed failed Slurm node daemon sshd.service loaded active running OpenSSH server daemon systemd-journal-catalog-update.service loaded active exited Rebuild Journal Catalog systemd-journal-flush.service loaded active exited Flush Journal to Persistent Storage systemd-journald.service loaded active running Journal Service systemd-sysusers.service loaded active exited Create System Users systemd-tmpfiles-setup.service loaded active exited Create Volatile Files and Directories systemd-update-done.service loaded active exited Update is Completed systemd-update-utmp.service loaded active exited Update UTMP about System Boot/Shutdown systemd-user-sessions.service loaded active exited Permit User Sessions -.slice loaded active active Root Slice [root@node07 ~]# systemctl slurmd status Unknown operation slurmd. [root@node07 ~]# systemctl status slurmd ● slurmd.service - Slurm node daemon Loaded: loaded (/usr/lib/systemd/system/slurmd.service; enabled; vendor preset: disabled) Drop-In: /usr/lib/systemd/system/slurmd.service.d └─cluster.conf /etc/systemd/system/slurmd.service.d └─local.conf Active: failed (Result: exit-code) since Fri 2024-01-12 04:14:29 EST; 7h ago Process: 121 ExecStart=/usr/local/sbin/slurmd --systemd $SLURMD_OPTIONS (code=exited, status=1/FAILURE) Process: 114 ExecStartPre=/usr/local/bin/slurmd.startup.sh (code=exited, status=0/SUCCESS) Process: 113 ExecStartPre=/usr/bin/chown slurm:slurm /var/log/slurmd.log (code=exited, status=0/SUCCESS) Process: 112 ExecStartPre=/usr/bin/touch /var/log/slurmd.log (code=exited, status=0/SUCCESS) Main PID: 121 (code=exited, status=1/FAILURE) Jan 12 04:14:29 node07 slurmd[121]: slurmd: error: ProctrackType 1 specified more than once, latest value used Jan 12 04:14:29 node07 slurmd[121]: slurmd: debug: Log file re-opened Jan 12 04:14:29 node07 slurmd[121]: slurmd: debug: CPUs:48 Boards:1 Sockets:2 CoresPerSocket:12 ThreadsPerCore:2 Jan 12 04:14:29 node07 slurmd[121]: slurmd: fatal: Hybrid mode is not supported. Mounted cgroups are: 9:devices:/ Jan 12 04:14:29 node07 slurmd[121]: 7:freezer:/ Jan 12 04:14:29 node07 slurmd[121]: 6:perf_event:/ Jan 12 04:14:29 node07 slurmd[121]: 5:net_cls,net_prio:/ Jan 12 04:14:29 node07 slurmd[121]: 1:name=systemd:/ Jan 12 04:14:29 node07 slurmd[121]: 0::/docker.slice/docker-07a88b0251dce9e294f193e08a3c5a8821a24e089f927876045b1542a43e1023.scope/init.scope [root@node07 ~]# systemctl restart slurmd Job for slurmd.service failed because the control process exited with error code. See "systemctl status slurmd.service" and "journalctl -xe" for details. [root@node07 ~]# journalctl -xe -- The system journal process has started up, opened the journal -- files for writing and is now ready to process requests. Jan 12 04:14:29 node07 systemd-journald[70]: Runtime journal (/run/log/journal/040e07a4495d48b1925d320d1aa78b73) is 8.0M, max 4.0G, 3.9G free. -- Subject: Disk space used by the journal -- Defined-By: systemd -- Support: https://access.redhat.com/support -- -- Runtime journal (/run/log/journal/040e07a4495d48b1925d320d1aa78b73) is currently using 8.0M. -- Maximum allowed usage is set to 4.0G. -- Leaving at least 4.0G free (of currently available 20.2G of disk space). -- Enforced usage limit is thus 4.0G, of which 3.9G are still available. -- -- The limits controlling how much disk space is used by the journal may -- be configured with SystemMaxUse=, SystemKeepFree=, SystemMaxFileSize=, -- RuntimeMaxUse=, RuntimeKeepFree=, RuntimeMaxFileSize= settings in -- /etc/systemd/journald.conf. See journald.conf(5) for details. Jan 12 04:14:29 node07 systemd-journald[70]: Runtime journal (/run/log/journal/040e07a4495d48b1925d320d1aa78b73) is 8.0M, max 4.0G, 3.9G free. -- Subject: Disk space used by the journal -- Defined-By: systemd -- Support: https://access.redhat.com/support -- -- Runtime journal (/run/log/journal/040e07a4495d48b1925d320d1aa78b73) is currently using 8.0M. -- Maximum allowed usage is set to 4.0G. -- Leaving at least 4.0G free (of currently available 20.2G of disk space). -- Enforced usage limit is thus 4.0G, of which 3.9G are still available. -- -- The limits controlling how much disk space is used by the journal may -- be configured with SystemMaxUse=, SystemKeepFree=, SystemMaxFileSize=, -- RuntimeMaxUse=, RuntimeKeepFree=, RuntimeMaxFileSize= settings in -- /etc/systemd/journald.conf. See journald.conf(5) for details. Jan 12 04:14:29 node07 systemd-tmpfiles[76]: [/usr/local/lib/tmpfiles.d/munge.conf:1] Line references path below legacy directory /var/run/, updating /var/run/munge → /run/munge; please update the tmpfiles.d/ d> Jan 12 04:14:29 node07 slurmd[121]: slurmd: error: ProctrackType 1 specified more than once, latest value used Jan 12 04:14:29 node07 slurmd[121]: slurmd: debug: Log file re-opened Jan 12 04:14:29 node07 slurmd[121]: slurmd: debug: CPUs:48 Boards:1 Sockets:2 CoresPerSocket:12 ThreadsPerCore:2 Jan 12 04:14:29 node07 slurmd[121]: slurmd: fatal: Hybrid mode is not supported. Mounted cgroups are: 9:devices:/ Jan 12 04:14:29 node07 slurmd[121]: 7:freezer:/ Jan 12 04:14:29 node07 slurmd[121]: 6:perf_event:/ Jan 12 04:14:29 node07 slurmd[121]: 5:net_cls,net_prio:/ Jan 12 04:14:29 node07 slurmd[121]: 1:name=systemd:/ Jan 12 04:14:29 node07 slurmd[121]: 0::/docker.slice/docker-07a88b0251dce9e294f193e08a3c5a8821a24e089f927876045b1542a43e1023.scope/init.scope Jan 12 11:51:24 node07 slurmd[258]: slurmd: error: ProctrackType 1 specified more than once, latest value used Jan 12 11:51:24 node07 slurmd[258]: slurmd: debug: Log file re-opened Jan 12 11:51:24 node07 slurmd[258]: slurmd: debug: CPUs:48 Boards:1 Sockets:2 CoresPerSocket:12 ThreadsPerCore:2 Jan 12 11:51:24 node07 slurmd[258]: slurmd: fatal: Hybrid mode is not supported. Mounted cgroups are: 9:devices:/ Jan 12 11:51:24 node07 slurmd[258]: 7:freezer:/ Jan 12 11:51:24 node07 slurmd[258]: 6:perf_event:/ Jan 12 11:51:24 node07 slurmd[258]: 5:net_cls,net_prio:/ Jan 12 11:51:24 node07 slurmd[258]: 1:name=systemd:/ Jan 12 11:51:24 node07 slurmd[258]: 0::/docker.slice/docker-07a88b0251dce9e294f193e08a3c5a8821a24e089f927876045b1542a43e1023.scope/init.scope ESCOC a78b73) is 8.0M, max 4.0G, 3.9G free. a78b73) is 8.0M, max 4.0G, 3.9G free. h below legacy directory /var/run/, updating /var/run/munge → /run/munge; please update the tmpfiles.d/ drop-in file accordingly. used erCore:2 evices:/ 876045b1542a43e1023.scope/init.scope used erCore:2 evices:/ 876045b1542a43e1023.scope/init.scope ESCOD -- The system journal process has started up, opened the journal -- files for writing and is now ready to process requests. Jan 12 04:14:29 node07 systemd-journald[70]: Runtime journal (/run/log/journal/040e07a4495d48b1925d320d1aa78b73) is 8.0M, max 4.0G, 3.9G free. -- Subject: Disk space used by the journal -- Defined-By: systemd -- Support: https://access.redhat.com/support -- -- Runtime journal (/run/log/journal/040e07a4495d48b1925d320d1aa78b73) is currently using 8.0M. -- Maximum allowed usage is set to 4.0G. -- Leaving at least 4.0G free (of currently available 20.2G of disk space). -- Enforced usage limit is thus 4.0G, of which 3.9G are still available. -- -- The limits controlling how much disk space is used by the journal may -- be configured with SystemMaxUse=, SystemKeepFree=, SystemMaxFileSize=, -- RuntimeMaxUse=, RuntimeKeepFree=, RuntimeMaxFileSize= settings in -- /etc/systemd/journald.conf. See journald.conf(5) for details. Jan 12 04:14:29 node07 systemd-journald[70]: Runtime journal (/run/log/journal/040e07a4495d48b1925d320d1aa78b73) is 8.0M, max 4.0G, 3.9G free. -- Subject: Disk space used by the journal -- Defined-By: systemd -- Support: https://access.redhat.com/support -- -- Runtime journal (/run/log/journal/040e07a4495d48b1925d320d1aa78b73) is currently using 8.0M. -- Maximum allowed usage is set to 4.0G. -- Leaving at least 4.0G free (of currently available 20.2G of disk space). -- Enforced usage limit is thus 4.0G, of which 3.9G are still available. -- -- The limits controlling how much disk space is used by the journal may -- be configured with SystemMaxUse=, SystemKeepFree=, SystemMaxFileSize=, -- RuntimeMaxUse=, RuntimeKeepFree=, RuntimeMaxFileSize= settings in -- /etc/systemd/journald.conf. See journald.conf(5) for details. Jan 12 04:14:29 node07 systemd-tmpfiles[76]: [/usr/local/lib/tmpfiles.d/munge.conf:1] Line references path below legacy directory /var/run/, updating /var/run/munge → /run/munge; please update the tmpfiles.d/ d> Jan 12 04:14:29 node07 slurmd[121]: slurmd: error: ProctrackType 1 specified more than once, latest value used Jan 12 04:14:29 node07 slurmd[121]: slurmd: debug: Log file re-opened Jan 12 04:14:29 node07 slurmd[121]: slurmd: debug: CPUs:48 Boards:1 Sockets:2 CoresPerSocket:12 ThreadsPerCore:2 Jan 12 04:14:29 node07 slurmd[121]: slurmd: fatal: Hybrid mode is not supported. Mounted cgroups are: 9:devices:/ Jan 12 04:14:29 node07 slurmd[121]: 7:freezer:/ Jan 12 04:14:29 node07 slurmd[121]: 6:perf_event:/ Jan 12 04:14:29 node07 slurmd[121]: 5:net_cls,net_prio:/ Jan 12 04:14:29 node07 slurmd[121]: 1:name=systemd:/ Jan 12 04:14:29 node07 slurmd[121]: 0::/docker.slice/docker-07a88b0251dce9e294f193e08a3c5a8821a24e089f927876045b1542a43e1023.scope/init.scope Jan 12 11:51:24 node07 slurmd[258]: slurmd: error: ProctrackType 1 specified more than once, latest value used Jan 12 11:51:24 node07 slurmd[258]: slurmd: debug: Log file re-opened Jan 12 11:51:24 node07 slurmd[258]: slurmd: debug: CPUs:48 Boards:1 Sockets:2 CoresPerSocket:12 ThreadsPerCore:2 Jan 12 11:51:24 node07 slurmd[258]: slurmd: fatal: Hybrid mode is not supported. Mounted cgroups are: 9:devices:/ Jan 12 11:51:24 node07 slurmd[258]: 7:freezer:/ Jan 12 11:51:24 node07 slurmd[258]: 6:perf_event:/ Jan 12 11:51:24 node07 slurmd[258]: 5:net_cls,net_prio:/ Jan 12 11:51:24 node07 slurmd[258]: 1:name=systemd:/ Jan 12 11:51:24 node07 slurmd[258]: 0::/docker.slice/docker-07a88b0251dce9e294f193e08a3c5a8821a24e089f927876045b1542a43e1023.scope/init.scope wilma's pam_slurm_adopt is not working (BTW, in our real cluster in el7 and el9 it does work, just reporting this with the test cluster I just built): [root@node09 /]# uname -a Linux node09 5.14.0-362.8.1.el9_3.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Nov 7 14:54:22 EST 2023 x86_64 x86_64 x86_64 GNU/Linux [root@node09 /]# ssh wilma@node01 wilma@node01's password: Access denied by pam_slurm_adopt: you have no active jobs on this node [wilma@node01 ~]$ netstat -pnat (No info could be read for "-p": geteuid()=1013 but you should be root.) Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN - tcp 0 0 127.0.0.11:40891 0.0.0.0:* LISTEN - tcp6 0 0 :::22 :::* LISTEN - tcp6 0 0 2001:db8:1:1::5:11:22 2001:db8:1:1::5:1:58838 ESTABLISHED - [wilma@node01 ~]$ route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface default _gateway 0.0.0.0 UG 0 0 0 eth0 10.11.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0 Main issue now for me to get slurmd working in node01-09. What could be the cause? Thanks!
(In reply to Serguei Mokhov from comment #15) > - It seems slurmd failed to start on each node[00-09] > - it appears the cluster added slurm_pam_adopt but ssh still allows through > - I see idle cloud[xxx] nodes?? I did not do a cloud build > - I can't ssh to the 'db' note and check it but can ping it Please open 1 ticket per issue as this ticket has gone quite a bit beyond the original issue which is still waiting review to inclusion upstream.
(In reply to Nate Rini from comment #16) > (In reply to Serguei Mokhov from comment #15) > > - It seems slurmd failed to start on each node[00-09] > > - it appears the cluster added slurm_pam_adopt but ssh still allows through > > - I see idle cloud[xxx] nodes?? I did not do a cloud build > > - I can't ssh to the 'db' note and check it but can ping it > > Please open 1 ticket per issue as this ticket has gone quite a bit beyond > the original issue which is still waiting review to inclusion upstream. Alright, will do, thank you! Sorry about that :)
(In reply to Serguei Mokhov from comment #17) > (In reply to Nate Rini from comment #16) > > (In reply to Serguei Mokhov from comment #15) > > > - It seems slurmd failed to start on each node[00-09] > > > - it appears the cluster added slurm_pam_adopt but ssh still allows through > > > - I see idle cloud[xxx] nodes?? I did not do a cloud build > > > - I can't ssh to the 'db' note and check it but can ping it > > > > Please open 1 ticket per issue as this ticket has gone quite a bit beyond > > the original issue which is still waiting review to inclusion upstream. > > Alright, will do, thank you! Sorry about that :) Opened: https://bugs.schedmd.com/show_bug.cgi?id=18664
Nate, related to this bug I just ran into another in the Makefile trying to make stop: https://github.com/naterini/docker-scale-out/pull/1
This has been fixed upstream on the master branch.
