SchedMD - Slurm Support – Full Text Ticket Listing
- Home
- | New
- | Browse
- | Search
- | [?]
- | Reports
- | Help
- | New Account
- | Log In
- | Forgot Password
| Summary: | 22.05 upgrade to 23.11 issues | ||
|---|---|---|---|
| Product: | Slurm | Reporter: | zohmanovic <zohmanovic> |
| Component: | slurmd | Assignee: | Jacob Jenson <jacob> |
| Status: | OPEN --- | QA Contact: | |
| Severity: | 6 - No support contract | ||
| Priority: | --- | ||
| Version: | 23.11.1 | ||
| Hardware: | Linux | ||
| OS: | Linux | ||
| Site: | -Other- | Slinky Site: | --- |
| Alineos Sites: | --- | Atos/Eviden Sites: | --- |
| Confidential Site: | --- | Coreweave sites: | --- |
| Cray Sites: | --- | DS9 clusters: | --- |
| Google sites: | --- | HPCnow Sites: | --- |
| HPE Sites: | --- | IBM Sites: | --- |
| NOAA SIte: | --- | NoveTech Sites: | --- |
| Nvidia HWinf-CS Sites: | --- | OCF Sites: | --- |
| Recursion Pharma Sites: | --- | SFW Sites: | --- |
| SNIC sites: | --- | Tzag Elita Sites: | --- |
| Linux Distro: | --- | Machine Name: | |
| CLE Version: | Version Fixed: | ||
| Target Release: | --- | DevPrio: | --- |
| Emory-Cloud Sites: | --- | ||
In addition job filled with: $ srun -p part1 --pty bash slurmstepd-host285: error: _get_joules_task: can't get info from slurmd (Slurm:52368718)user@host285[~]$ (Slurm:52368718)user@host285[~]$ (Slurm:52368718)user@host285[~]$ slurmstepd-host285: error: _get_joules_task: can't get info from slurmd slurmstepd-host285: error: _get_joules_task: can't get info from slurmd slurmstepd-host285: error: _get_joules_task: can't get info from slurmd .. .. Update: Currntly downgraded slurmd daemons to 22.05, dbd and control run in 23.11, looks stable now. |
Hi, We're trying to upgrade to 23.11 but having several issues, hopfully you can help. The main change was changing --systemd + Type=notify argument on the service instead of -D and Type=simple. /usr/local/lib/systemd/system/slurmd.service .. [Service] Type=notify EnvironmentFile=-/etc/default/slurmd RuntimeDirectory=slurm RuntimeDirectoryMode=0755 ExecStart=/usr/nas/pkgs/slurm/23.11/sbin/slurmd --systemd -v ExecReload=/bin/kill -HUP $MAINPID KillMode=process LimitNOFILE=131072 LimitMEMLOCK=infinity LimitSTACK=infinity TasksMax=infinity Delegate=yes .. after some time slurmd seems to throw many: slurmd-host409: error: _rpc_acct_gather_energy: Too many pending requests slurmd-host409: error: _rpc_acct_gather_energy: Too many pending requests slurmd-host409: error: _rpc_acct_gather_energy: Too many pending requests slurmd-host409: error: _rpc_acct_gather_energy: Too many pending requests slurmd-host409: error: _rpc_acct_gather_energy: Too many pending requests this and some machines open duplicate daemons: ● slurmd.service - Slurm node daemon Loaded: loaded (/usr/local/lib/systemd/system/slurmd.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2024-01-02 12:47:15 IST; 1h 25min ago Docs: man:slurmd(8) Main PID: 2592518 (slurmd) Tasks: 11 Memory: 209.5M CPU: 1min 24.861s CGroup: /system.slice/slurmd.service ├─2278264 /usr/nas/pkgs/slurm/23.11/sbin/slurmd --systemd -v ├─2278280 /usr/nas/pkgs/slurm/23.11/sbin/slurmd --systemd -v ├─2278291 /usr/nas/pkgs/slurm/23.11/sbin/slurmd --systemd -v ├─2380536 /usr/nas/pkgs/slurm/23.11/sbin/slurmd --systemd -v ├─2478980 /usr/nas/pkgs/slurm/23.11/sbin/slurmd --systemd -v ├─2592518 /usr/nas/pkgs/slurm/23.11/sbin/slurmd --systemd -v ├─2791550 /usr/nas/pkgs/slurm/23.11/sbin/slurmd --systemd -v ├─2791572 /usr/nas/pkgs/slurm/23.11/sbin/slurmd --systemd -v ├─2791582 /usr/nas/pkgs/slurm/23.11/sbin/slurmd --systemd -v └─2791604 /usr/nas/pkgs/slurm/23.11/sbin/slurmd --systemd -v restarting such service fail with: systemd[1]: slurmd.service: Failed with result 'exit-code'. systemd[1]: slurmd.service: Unit process 4165811 (slurmd) remains running after unit stopped. systemd[1]: slurmd.service: Unit process 4165844 (slurmd) remains running after unit stopped. systemd[1]: slurmd.service: Unit process 4165850 (slurmd) remains running after unit stopped. systemd[1]: slurmd.service: Unit process 4165878 (slurmd) remains running after unit stopped. systemd[1]: slurmd.service: Unit process 4165889 (slurmd) remains running after unit stopped. systemd[1]: slurmd.service: Unit process 82813 (slurmd) remains running after unit stopped. systemd[1]: slurmd.service: Unit process 82836 (slurmd) remains running after unit stopped. systemd[1]: slurmd.service: Unit process 82865 (slurmd) remains running after unit stopped. there is no issue to allocate jobs, but now overtime nodes become down.. # sinfo -R Not responding root 14:04:39-02/01/24 host154 Not responding root 14:17:59-02/01/24 host273 and also pam_slurm_adopt stop working.. pam_slurm_adopt[2132106]: fatal: No hash plugins loaded. Was slurm_init() called before calling any Slurm API functions? slurmd[2842699]: slurmd-host159: error: slurm_receive_msg_and_forward: [[host159.example.com]:50314] failed: Zero Bytes were transmitted or received what I'm missing?