Ticket 16376 - Slurm logs appear in /var/log/message
Summary: Slurm logs appear in /var/log/message
Status: RESOLVED INFOGIVEN
Alias: None
Product: Slurm
Classification: Unclassified
Component: slurmctld (show other tickets)
Version: 22.05.8
Hardware: Linux Linux
: 4 - Minor Issue
Assignee: Tim McMullan
QA Contact:
URL:
Depends on:
Blocks:
 
Reported: 2023-03-27 14:25 MDT by Misha Ahmadian
Modified: 2023-03-29 12:55 MDT (History)
0 users

See Also:
Site: TTU
Slinky Site: ---
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
Google sites: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
NoveTech Sites: ---
Nvidia HWinf-CS Sites: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Tzag Elita Sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed:
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---

Attachments
slurm.conf (6.72 KB, text/plain)
2023-03-27 14:28 MDT, Misha Ahmadian 		Details
slurmctld.service (611 bytes, text/x-systemd-unit)
2023-03-27 15:38 MDT, Misha Ahmadian 		Details
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this ticket.
Description Misha Ahmadian 2023-03-27 14:25:29 MDT 
Hello,

We've noticed that all the logs in the slurmctld.log file also appear in the same order in /var/log/message since we moved from v20.11 to v22.5. Is there any missing configuration in our slurm.conf file that is supposed to control this behavior?

Best Regards,
Misha
Comment 1 Misha Ahmadian 2023-03-27 14:28:07 MDT 
Created attachment 29542 [details]
slurm.conf
Comment 2 Jason Booth 2023-03-27 15:33:32 MDT 
Can you also attach/copy into this ticket the systemd init script used to start slurmctld?
Comment 3 Misha Ahmadian 2023-03-27 15:38:35 MDT 
Created attachment 29546 [details]
slurmctld.service

Hi Jason,

Please find the attached slurmctld.service copied from /usr/lib/systemd/system/slurmctld.service. 

PS: The "/etc/sysconfig/slurmctld" doesn't exist in our settings.

Best Regards,
Misha
Comment 4 Tim McMullan 2023-03-29 06:57:56 MDT 
Hi Misha,

There was a bug fixed in slurm 21.08 that prevented us from logging to a socket.  After this change, systemd-journald now gets the logs as well as the log file... however systemd-journald can redirect those logs elsewhere causing the logs to show up in both the log file and in /var/log/messages.

If this is a problem for you we recommend the following change to prevent the logs from making it into systemd-journald.

You can try creating an override file for the slurmctld unit that looks like this:
> $ cat /etc/systemd/system/slurmctld.service.d/slurmctld-logging.conf
> [Service]
> StandardOutput=null
> StandardError=null

After creating that file, you'd need to run "systemctl daemon-reload" and likely restart the slurmctld for it to take effect.  This should prevent the logs from making it into systemd-journald and wherever it may redirect those logs.

We suggest using override files like this so that the change will persist across slurm upgrades.

Let me know if this helps!
--Tim
Comment 5 Misha Ahmadian 2023-03-29 12:55:31 MDT 
Hi Tim,

Thank you very much. That helped fix the issue here!

Best Regards,
Misha