Ticket 3933 - Send log messages to both syslog and the log file
Summary: Send log messages to both syslog and the log file
Status: RESOLVED FIXED
Alias: None
Product: Slurm
Classification: Unclassified
Component: slurmctld (show other tickets)
Version: 17.11.x
Hardware: Linux Linux
: 5 - Enhancement
Assignee: Director of Support
QA Contact:
URL:
Depends on:
Blocks:
 
Reported: 2017-06-28 01:53 MDT by Tim Wickberg
Modified: 2017-07-26 13:32 MDT (History)
4 users (show)

See Also:
Site: LANL
Slinky Site: ---
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
Google sites: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
NoveTech Sites: ---
Nvidia HWinf-CS Sites: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Tzag Elita Sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed: 17.11.0-pre2
Target Release: 17.11
DevPrio: ---
Emory-Cloud Sites: ---

Attachments
Syslog config options added (10.87 KB, patch)
2017-07-20 16:00 MDT, Isaac Hartung 		Details | Diff
logs flow to syslog and logfile at specified detail. (18.90 KB, patch)
2017-07-25 14:23 MDT, Isaac Hartung 		Details | Diff
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this ticket.
Description Tim Wickberg 2017-06-28 01:53:30 MDT 
LANL would like to have their logs flow both to the local logfile, and to syslog to be captured on an external system.

Currently, syslog will only receive messages at LOG_LEVEL_FATAL if the SlurmctldLogFile is defined.

The logging code appears to be capable of duplicating the messages without issue if log_opts.syslog_level is set appropriately, so it looks like the only real issue here would be deciding how to add an option to control this.

I propose adding separate SlurmctldSyslogDebug / SlurmdSyslogDebug options to slurm.conf, and DebugLevelSyslog to slurmdbd.conf to handle setting this. If those options are not explicitly set, continue to use the current behavior.
Comment 3 Isaac Hartung 2017-07-20 16:00:24 MDT 
Created attachment 4947 [details]
Syslog config options added
Comment 4 Moe Jette 2017-07-21 11:38:54 MDT 
(In reply to Isaac Hartung from comment #3)
> Created attachment 4947 [details]
> Syslog config options added

That's a partial solution.

SlurmctldSyslogDebug and the other configuration parameters should be changed
from yes/no values to taking the same options as SlurmctldDebug ("fatal",
"error", "info", etc.). That provides the greatest flexibility. For example, some sites might just want to write fatal errors to syslog, while others what to write all of the "info" type messages.

Similarly, change the interval values from type "char *" to "uint16_t".
That number should be fed directly into "log_opts.syslog_level" for the log
functions.

A couple of lines in "src/common/read_config.c" need to be reformatted to stay
within 80 characters.

These new fields need to be made visible to "scontrol show config" and sview.
That requires:
1. Adding new fields to "slurm_ctl_conf_t" in "slurm/slurm.h.in" (remember to
   execute "configure" after updating the file.
2. Populating the data structure from slurmctld. See "_fill_ctld_conf" in
   src/slurmctld/proc_req.c
3. Including the new information in the return message. See
   _pack_slurm_ctl_conf_msg() and _unpack_slurm_ctl_conf_msg() in
   src/common/slurm_protocol_pack.c. Only change the RPC for version 17.11 of
   the un/pack functions. The data does not exist in earlier versions, so
   nothing is appropriate to un/pack.
4. Print the data. See slurm_write_ctl_conf() and slurm_ctl_conf_2_key_pairs()
   in src/api/config_info.c

Validate that everything works and you should be all set.
Comment 5 Isaac Hartung 2017-07-25 14:23:48 MDT 
Created attachment 4961 [details]
logs flow to syslog and logfile at specified detail.
Comment 7 Moe Jette 2017-07-26 13:32:24 MDT 
Isaac is on vacation right now, slightly modified version of is patch has been committed to version 17.11:

https://github.com/SchedMD/slurm/commit/05ee90f129b1b0d37eb9567d8c486104c2698210