Ticket 16672

Hi,

Just for testing purposes, can you please try restarting the controller?  with the slurmctld command?  And you confirm that without adding the lua script there a re no issues?

Thanks

Running slurmctld manually still fails.

nid000008:~ # /usr/sbin/slurmctld -D -s
slurmctld: debug:  slurmctld log levels: stderr=debug logfile=debug syslog=quiet
slurmctld: debug:  Log file re-opened
slurmctld: pidfile not locked, assuming no running daemon
slurmctld: error: Configured MailProg is invalid
slurmctld: debug:  slurmscriptd: Got ack from slurmctld
slurmctld: select/cons_res: common_init: select/cons_res loaded
slurmctld: select/cons_tres: common_init: select/cons_tres loaded
slurmctld: select/cray_aries: init: Cray/Aries node selection plugin loaded
slurmctld: select/linear: init: Linear node selection plugin loaded with argument 20
slurmctld: debug:  Initialization successful
slurmctld: debug:  _slurmscriptd_mainloop: started
slurmctld: debug:  slurmctld: slurmscriptd fork()'d and initialized.
slurmctld: debug:  _slurmctld_listener_thread: started listening to slurmscriptd
slurmctld: slurmctld version 23.02.1 started on cluster sawmill
slurmctld: cred/munge: init: Munge credential signature plugin loaded
slurmctld: accounting_storage/none: init: Accounting storage NOT INVOKED plugin loaded
slurmctld: select/cons_res: common_init: select/cons_res loaded
slurmctld: select/cons_tres: common_init: select/cons_tres loaded
slurmctld: select/cray_aries: init: Cray/Aries node selection plugin loaded
slurmctld: select/linear: init: Linear node selection plugin loaded with argument 20
slurmctld: preempt/none: init: preempt/none loaded
slurmctld: debug:  acct_gather_energy/none: init: AcctGatherEnergy NONE plugin loaded
slurmctld: debug:  acct_gather_profile/none: init: AcctGatherProfile NONE plugin loaded
slurmctld: debug:  acct_gather_interconnect/none: init: AcctGatherInterconnect NONE plugin loaded
slurmctld: debug:  acct_gather_filesystem/none: init: AcctGatherFilesystem NONE plugin loaded
slurmctld: debug:  jobacct_gather/none: init: Job accounting gather NOT_INVOKED plugin loaded
slurmctld: debug:  _slurmscriptd_mainloop: finished
Segmentation fault

If I comment out JobSubmitPlugins=lua from slurm.conf, slurmctld starts fine.

nid000008:~ # systemctl status slurmctld
● slurmctld.service - Slurm controller daemon
     Loaded: loaded (/usr/lib/systemd/system/slurmctld.service; disabled; vendor preset: disabled)
     Active: active (running) since Tue 2023-05-09 15:58:29 UTC; 55s ago
   Main PID: 113516 (slurmctld)
      Tasks: 11
     CGroup: /system.slice/slurmctld.service
             ├─ 113516 /usr/sbin/slurmctld -D -s
             └─ 113517 "slurmctld: slurmscriptd" "" ""

May 09 15:58:29 nid000008 slurmctld[113516]: slurmctld: debug:  No backup controllers, not launching heart>
May 09 15:58:29 nid000008 slurmctld[113516]: slurmctld: No parameter for mcs plugin, default values set
May 09 15:58:29 nid000008 slurmctld[113516]: slurmctld: mcs: MCSParameters = (null). ondemand set.
May 09 15:58:29 nid000008 slurmctld[113516]: slurmctld: debug:  mcs/none: init: mcs none plugin loaded
May 09 15:58:33 nid000008 slurmctld[113516]: slurmctld: debug:  Spawning registration agent for nid[000008>
May 09 15:58:44 nid000008 slurmctld[113516]: slurmctld: agent/is_node_resp: node:nid000008 RPC:REQUEST_NOD>
May 09 15:58:44 nid000008 slurmctld[113516]: slurmctld: agent/is_node_resp: node:nid000009 RPC:REQUEST_NOD>
May 09 15:58:45 nid000008 slurmctld[113516]: slurmctld: error: Nodes nid[000008-000009] not responding
May 09 15:58:59 nid000008 slurmctld[113516]: slurmctld: debug:  sched/backfill: _attempt_backfill: beginni>
May 09 15:58:59 nid000008 slurmctld[113516]: slurmctld: debug:  sched/backfill: _attempt_backfill: no jobs>

Thanks for sharing that.  Can you please attach a copy of your slurm.conf?

Regards

Created attachment 30187 [details]
slurm.conf file

Also, did you follow the setup instructions listed here:

https://slurm.schedmd.com/job_submit_plugins.html

?

Thanks

We just copied the example job_submit plugin, assuming that the developer of the example plugin followed that guide.

Ok, can you tell me what lua rpm you have installed?

Thanks

nid000008:~ # rpm -qa | grep lua
lua-macros-20170611-1.152.noarch
liblua5_3-5-5.3.6-3.6.1.x86_64
liblua5_1-5-5.1.5-2.31.x86_64
lua51-devel-5.1.5-2.31.x86_64
lua51-5.1.5-2.31.x86_64

From what I can gather, you built with lua 5.1 headers, but the lua 5.3 library is being loaded.

You should most likely remove lua5.1-devel and install lua5.3-devel.

Please let me know if that helps.

Thanks

Hi

After further reflection and analysis, I think you would benefit from installing the latest lua version altogether and making sure the latest development files are installed.  If you haven't already done so, try holding off from removing the 5.1 version for now.   

Thanks

Hi,

Did you manage to make any headway regarding this issue?

Thanks

The development system where we originally hit this issue is currently busy. Once it's available again we plan on installing lua53 and lua53-devel instead of lua51 and lua51-devel.

Hi,

Ok sounds good.  I will keep this ticket open if you'd like until the issue is resolved.  Unless you foresee this taking some time, in that case we can close and you can re-open if you experience more issues.

Thanks

Installing lua53 and lua53-devel fixed this problem, so I think this can be closed.

Hi,

That's great!  I'm glad you got it resolved.  

Have a great day