Ticket 6732 - Improve slurmctld threads/agents tracking
Summary: Improve slurmctld threads/agents tracking
Status: OPEN
Alias: None
Product: Slurm
Classification: Unclassified
Component: slurmctld (show other tickets)
Version: 19.05.x
Hardware: Linux Linux
: 5 - Enhancement
Assignee: Alejandro Sanchez
QA Contact:
URL:
: 7501 (view as ticket list)
Depends on:
Blocks:
 
Reported: 2019-03-20 11:13 MDT by Alejandro Sanchez
Modified: 2020-08-27 03:50 MDT (History)
3 users (show)

See Also:
Site: SchedMD
Slinky Site: ---
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
Google sites: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
NoveTech Sites: ---
Nvidia HWinf-CS Sites: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Tzag Elita Sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed:
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this ticket.
Description Alejandro Sanchez 2019-03-20 11:13:52 MDT 
slurmctld maintains a separate count of incoming message traffic in slurmctld_config.server_thread_count and outgoing message traffic in agent_cnt / agent_thread_cnt. 

Then ctld defers certain operations if the former remains below certain thresholds or the latter remains below others, but in some places, it might be desirable to observe if the sum of both incoming/outgoing remains below a reasonable number.

Background discussion:

https://bugs.schedmd.com/show_bug.cgi?id=5443#c69

and

https://bugs.schedmd.com/show_bug.cgi?id=5443#c170

and

https://bugs.schedmd.com/show_bug.cgi?id=6189#c10
Comment 4 Nate Rini 2019-08-28 18:18:24 MDT 
*** Ticket 7501 has been marked as a duplicate of this ticket. ***
Comment 5 Alejandro Sanchez 2020-08-27 03:50:18 MDT 
After letting sites breath for a while with 2b25aa4e555 and 67a9c2f786e8 it looks like things got better, so the rest of the proposed ideas can be reclassified as sev-5.