Ticket 18561 - 22.05 upgrade to 23.11 issues
Summary: 22.05 upgrade to 23.11 issues
Status: OPEN
Alias: None
Product: Slurm
Classification: Unclassified
Component: slurmd (show other tickets)
Version: 23.11.1
Hardware: Linux Linux
: 6 - No support contract
Assignee: Jacob Jenson
QA Contact:
URL:
Depends on:
Blocks:
 
Reported: 2024-01-02 05:39 MST by zohmanovic@gmail.com
Modified: 2024-01-24 09:10 MST (History)
0 users

See Also:
Site: -Other-
Slinky Site: ---
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
Google sites: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
NoveTech Sites: ---
Nvidia HWinf-CS Sites: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Tzag Elita Sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed:
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this ticket.
Description zohmanovic@gmail.com 2024-01-02 05:39:30 MST 
Hi,

We're trying to upgrade to 23.11 but having several issues, hopfully you can help.


The main change was changing --systemd + Type=notify argument on the service instead of -D and Type=simple.

/usr/local/lib/systemd/system/slurmd.service
..
[Service]
Type=notify
EnvironmentFile=-/etc/default/slurmd
RuntimeDirectory=slurm
RuntimeDirectoryMode=0755
ExecStart=/usr/nas/pkgs/slurm/23.11/sbin/slurmd --systemd -v
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
LimitNOFILE=131072
LimitMEMLOCK=infinity
LimitSTACK=infinity
TasksMax=infinity
Delegate=yes
..

after some time slurmd seems to throw many:

slurmd-host409: error: _rpc_acct_gather_energy: Too many pending requests
slurmd-host409: error: _rpc_acct_gather_energy: Too many pending requests
slurmd-host409: error: _rpc_acct_gather_energy: Too many pending requests
slurmd-host409: error: _rpc_acct_gather_energy: Too many pending requests
slurmd-host409: error: _rpc_acct_gather_energy: Too many pending requests

this and some machines open duplicate daemons:

● slurmd.service - Slurm node daemon
     Loaded: loaded (/usr/local/lib/systemd/system/slurmd.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2024-01-02 12:47:15 IST; 1h 25min ago
       Docs: man:slurmd(8)
   Main PID: 2592518 (slurmd)
      Tasks: 11
     Memory: 209.5M
        CPU: 1min 24.861s
     CGroup: /system.slice/slurmd.service
             ├─2278264 /usr/nas/pkgs/slurm/23.11/sbin/slurmd --systemd -v
             ├─2278280 /usr/nas/pkgs/slurm/23.11/sbin/slurmd --systemd -v
             ├─2278291 /usr/nas/pkgs/slurm/23.11/sbin/slurmd --systemd -v
             ├─2380536 /usr/nas/pkgs/slurm/23.11/sbin/slurmd --systemd -v
             ├─2478980 /usr/nas/pkgs/slurm/23.11/sbin/slurmd --systemd -v
             ├─2592518 /usr/nas/pkgs/slurm/23.11/sbin/slurmd --systemd -v
             ├─2791550 /usr/nas/pkgs/slurm/23.11/sbin/slurmd --systemd -v
             ├─2791572 /usr/nas/pkgs/slurm/23.11/sbin/slurmd --systemd -v
             ├─2791582 /usr/nas/pkgs/slurm/23.11/sbin/slurmd --systemd -v
             └─2791604 /usr/nas/pkgs/slurm/23.11/sbin/slurmd --systemd -v

restarting such service fail with:

systemd[1]: slurmd.service: Failed with result 'exit-code'.
systemd[1]: slurmd.service: Unit process 4165811 (slurmd) remains running after unit stopped.
systemd[1]: slurmd.service: Unit process 4165844 (slurmd) remains running after unit stopped.
systemd[1]: slurmd.service: Unit process 4165850 (slurmd) remains running after unit stopped.
systemd[1]: slurmd.service: Unit process 4165878 (slurmd) remains running after unit stopped.
systemd[1]: slurmd.service: Unit process 4165889 (slurmd) remains running after unit stopped.
systemd[1]: slurmd.service: Unit process 82813 (slurmd) remains running after unit stopped.
systemd[1]: slurmd.service: Unit process 82836 (slurmd) remains running after unit stopped.
systemd[1]: slurmd.service: Unit process 82865 (slurmd) remains running after unit stopped.


there is no issue to allocate jobs, but now overtime nodes become down..

# sinfo -R
Not responding       root      14:04:39-02/01/24   host154
Not responding       root      14:17:59-02/01/24   host273

and also pam_slurm_adopt stop working..

pam_slurm_adopt[2132106]: fatal: No hash plugins loaded. Was slurm_init() called before calling any Slurm API functions?
slurmd[2842699]: slurmd-host159: error: slurm_receive_msg_and_forward: [[host159.example.com]:50314] failed: Zero Bytes were transmitted or received



what I'm missing?
Comment 1 zohmanovic@gmail.com 2024-01-02 06:10:43 MST 
In addition job filled with:

$ srun -p part1 --pty bash
slurmstepd-host285: error: _get_joules_task: can't get info from slurmd
(Slurm:52368718)user@host285[~]$ 
(Slurm:52368718)user@host285[~]$ 
(Slurm:52368718)user@host285[~]$ slurmstepd-host285: error: _get_joules_task: can't get info from slurmd
slurmstepd-host285: error: _get_joules_task: can't get info from slurmd
slurmstepd-host285: error: _get_joules_task: can't get info from slurmd
..
..
Comment 3 zohmanovic@gmail.com 2024-01-07 04:18:08 MST 
Update: Currntly downgraded slurmd daemons to 22.05,
dbd and control run in 23.11, looks stable now.