Ticket 2781

Summary: Can JobAcctGatherType be safely changed when running jobs in production?
Product: Slurm Reporter: Chris Samuel <samuel>
Component: ConfigurationAssignee: Alejandro Sanchez <asanchezg>
Status: RESOLVED INFOGIVEN QA Contact:
Severity: 4 - Minor Issue    
Priority: --- CC: asanchezg
Version: 15.08.8   
Hardware: Linux   
OS: Linux   
Site: VLSCI Slinky Site: ---
Alineos Sites: --- Atos/Eviden Sites: ---
Confidential Site: --- Coreweave sites: ---
Cray Sites: --- DS9 clusters: ---
Google sites: --- HPCnow Sites: ---
HPE Sites: --- IBM Sites: ---
NOAA SIte: --- NoveTech Sites: ---
Nvidia HWinf-CS Sites: --- OCF Sites: ---
Recursion Pharma Sites: --- SFW Sites: ---
SNIC sites: --- Tzag Elita Sites: ---
Linux Distro: --- Machine Name:
CLE Version: Version Fixed:
Target Release: --- DevPrio: ---
Emory-Cloud Sites: ---

Description Chris Samuel 2016-05-30 13:05:34 MDT 
Hi there,

Trying to catch up on a few things here, Danny recommended we switch JobAcctGatherType from jobacct_gather/cgroup to jobacct_gather/linux.

Is that something that can be safely updated whilst Slurm is running or will the slurmd's be unhappy to swap how they do things on the fly?

All the best,
Chris
Comment 1 Alejandro Sanchez 2016-05-30 19:16:41 MDT 
Hi Chris. Unfortunately this parameter can't be modified on the fly with just 'scontrol reconfigure'. A slurmd restart while there are no running job steps in the node is required. Copying from slurm.conf man page a footnote on a change to this parameter:

NOTE: Changing this configuration parameter changes the contents of the messages  between Slurm daemons. Any previously running job steps are managed by a slurmstepd daemon that will persist through the lifetime of that job step and not change it's communication protocol. Only change this configuration parameter when there are no running job steps.
Comment 2 Chris Samuel 2016-05-30 19:21:43 MDT 
(In reply to Alejandro Sanchez from comment #1)

> Hi Chris. Unfortunately this parameter can't be modified on the fly with
> just 'scontrol reconfigure'. A slurmd restart while there are no running job
> steps in the node is required. Copying from slurm.conf man page a footnote
> on a change to this parameter:

Thanks & so sorry I missed that in the man page before! :-(

OK, so basically we'll need to put this off until we can take an outage.

Thanks!