Ticket 3933

Summary:	Send log messages to both syslog and the log file
Product:	Slurm	Reporter:	Tim Wickberg <tim>
Component:	slurmctld	Assignee:	Director of Support <support>
Status:	RESOLVED FIXED	QA Contact:
Severity:	5 - Enhancement
Priority:	---	CC:	alex, mej, peltzpl, sts
Version:	17.11.x
Hardware:	Linux
OS:	Linux
Site:	LANL	Alineos Sites:	---
Atos/Eviden Sites:	---	Confidential Site:	---
Coreweave sites:	---	Cray Sites:	---
DS9 clusters:	---	HPCnow Sites:	---
HPE Sites:	---	IBM Sites:	---
NOAA SIte:	---	OCF Sites:	---
Recursion Pharma Sites:	---	SFW Sites:	---
SNIC sites:	---	Linux Distro:	---
Machine Name:		CLE Version:
Version Fixed:	17.11.0-pre2	Target Release:	17.11
DevPrio:	---	Emory-Cloud Sites:	---
Attachments:	Syslog config options added logs flow to syslog and logfile at specified detail.

Description Tim Wickberg 2017-06-28 01:53:30 MDT

LANL would like to have their logs flow both to the local logfile, and to syslog to be captured on an external system.

Currently, syslog will only receive messages at LOG_LEVEL_FATAL if the SlurmctldLogFile is defined.

The logging code appears to be capable of duplicating the messages without issue if log_opts.syslog_level is set appropriately, so it looks like the only real issue here would be deciding how to add an option to control this.

I propose adding separate SlurmctldSyslogDebug / SlurmdSyslogDebug options to slurm.conf, and DebugLevelSyslog to slurmdbd.conf to handle setting this. If those options are not explicitly set, continue to use the current behavior.

Comment 3 Isaac Hartung 2017-07-20 16:00:24 MDT

Created attachment 4947 [details]
Syslog config options added

Comment 4 Moe Jette 2017-07-21 11:38:54 MDT

(In reply to Isaac Hartung from comment #3)
> Created attachment 4947 [details]
> Syslog config options added

That's a partial solution.

SlurmctldSyslogDebug and the other configuration parameters should be changed
from yes/no values to taking the same options as SlurmctldDebug ("fatal",
"error", "info", etc.). That provides the greatest flexibility. For example, some sites might just want to write fatal errors to syslog, while others what to write all of the "info" type messages.

Similarly, change the interval values from type "char *" to "uint16_t".
That number should be fed directly into "log_opts.syslog_level" for the log
functions.

A couple of lines in "src/common/read_config.c" need to be reformatted to stay
within 80 characters.

These new fields need to be made visible to "scontrol show config" and sview.
That requires:
1. Adding new fields to "slurm_ctl_conf_t" in "slurm/slurm.h.in" (remember to
   execute "configure" after updating the file.
2. Populating the data structure from slurmctld. See "_fill_ctld_conf" in
   src/slurmctld/proc_req.c
3. Including the new information in the return message. See
   _pack_slurm_ctl_conf_msg() and _unpack_slurm_ctl_conf_msg() in
   src/common/slurm_protocol_pack.c. Only change the RPC for version 17.11 of
   the un/pack functions. The data does not exist in earlier versions, so
   nothing is appropriate to un/pack.
4. Print the data. See slurm_write_ctl_conf() and slurm_ctl_conf_2_key_pairs()
   in src/api/config_info.c

Validate that everything works and you should be all set.

Comment 5 Isaac Hartung 2017-07-25 14:23:48 MDT

Created attachment 4961 [details]
logs flow to syslog and logfile at specified detail.

Comment 7 Moe Jette 2017-07-26 13:32:24 MDT

Isaac is on vacation right now, slightly modified version of is patch has been committed to version 17.11:

https://github.com/SchedMD/slurm/commit/05ee90f129b1b0d37eb9567d8c486104c2698210