Ticket 16376

Summary:	Slurm logs appear in /var/log/message
Product:	Slurm	Reporter:	Misha Ahmadian <misha.ahmadian>
Component:	slurmctld	Assignee:	Tim McMullan <mcmullan>
Status:	RESOLVED INFOGIVEN	QA Contact:
Severity:	4 - Minor Issue
Priority:	---
Version:	22.05.8
Hardware:	Linux
OS:	Linux
Site:	TTU	Alineos Sites:	---
Atos/Eviden Sites:	---	Confidential Site:	---
Coreweave sites:	---	Cray Sites:	---
DS9 clusters:	---	HPCnow Sites:	---
HPE Sites:	---	IBM Sites:	---
NOAA SIte:	---	OCF Sites:	---
Recursion Pharma Sites:	---	SFW Sites:	---
SNIC sites:	---	Linux Distro:	---
Machine Name:		CLE Version:
Version Fixed:		Target Release:	---
DevPrio:	---	Emory-Cloud Sites:	---
Attachments:	slurm.conf slurmctld.service

Description Misha Ahmadian 2023-03-27 14:25:29 MDT

Hello,

We've noticed that all the logs in the slurmctld.log file also appear in the same order in /var/log/message since we moved from v20.11 to v22.5. Is there any missing configuration in our slurm.conf file that is supposed to control this behavior?

Best Regards,
Misha

Comment 1 Misha Ahmadian 2023-03-27 14:28:07 MDT

Created attachment 29542 [details]
slurm.conf

Comment 2 Jason Booth 2023-03-27 15:33:32 MDT

Can you also attach/copy into this ticket the systemd init script used to start slurmctld?

Comment 3 Misha Ahmadian 2023-03-27 15:38:35 MDT

Created attachment 29546 [details]
slurmctld.service

Hi Jason,

Please find the attached slurmctld.service copied from /usr/lib/systemd/system/slurmctld.service. 

PS: The "/etc/sysconfig/slurmctld" doesn't exist in our settings.

Best Regards,
Misha

Comment 4 Tim McMullan 2023-03-29 06:57:56 MDT

Hi Misha,

There was a bug fixed in slurm 21.08 that prevented us from logging to a socket.  After this change, systemd-journald now gets the logs as well as the log file... however systemd-journald can redirect those logs elsewhere causing the logs to show up in both the log file and in /var/log/messages.

If this is a problem for you we recommend the following change to prevent the logs from making it into systemd-journald.

You can try creating an override file for the slurmctld unit that looks like this:
> $ cat /etc/systemd/system/slurmctld.service.d/slurmctld-logging.conf
> [Service]
> StandardOutput=null
> StandardError=null

After creating that file, you'd need to run "systemctl daemon-reload" and likely restart the slurmctld for it to take effect.  This should prevent the logs from making it into systemd-journald and wherever it may redirect those logs.

We suggest using override files like this so that the change will persist across slurm upgrades.

Let me know if this helps!
--Tim

Comment 5 Misha Ahmadian 2023-03-29 12:55:31 MDT

Hi Tim,

Thank you very much. That helped fix the issue here!

Best Regards,
Misha