Ticket 8325 - Missing timeouts for elasticsearch plugin lead to stuck controller
Summary: Missing timeouts for elasticsearch plugin lead to stuck controller
Status: RESOLVED FIXED
Alias: None
Product: Slurm
Classification: Unclassified
Component: slurmctld (show other tickets)
Version: 20.02.x
Hardware: Linux Linux
: C - Contributions
Assignee: Nate Rini
QA Contact: Tim Wickberg
URL:
: 8379 (view as ticket list)
Depends on:
Blocks:
 
Reported: 2020-01-13 06:28 MST by Regine Gaudin
Modified: 2022-02-23 10:39 MST (History)
2 users (show)

See Also:
Site: CEA
Slinky Site: ---
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
Google sites: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
NoveTech Sites: ---
Nvidia HWinf-CS Sites: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Tzag Elita Sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed: 20.02
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---

Attachments
use two env variables for curl timeout (3.37 KB, patch)
2020-01-13 06:28 MST, Regine Gaudin 		Details | Diff
slurm-18.08.6-elasticsearch.patch (3.78 KB, patch)
2020-01-23 03:13 MST, Regine Gaudin 		Details | Diff
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this ticket.
Description Regine Gaudin 2020-01-13 06:28:04 MST 
Created attachment 12715 [details]
use two env variables for curl timeout

We are using elasticsearch plugin with slurm.18. The http server did not respond after a oom killer and this problem leads to stuck slurmctld infinite wait.

Alghough we need to increase robustness and ha of http server, give a  possibility of setting timeouts in slurm.conf for plugin jobcomp/elasticsearch 
would prevent such occurance. Is it possible for schedmd to do it ?

As it means to add members to slurm_ctl_conf_t structure it is better if schedmd 
implements it. 

A "less clean" war consisting in adding environment variables read in jobcomp_elasticsearch.c and used in curl_set_opt(CURL_TIMEOUT) , curl_set_opt(CURL_CONNECTTIMEOUT) before curl_easy_perform have been used.
See in atatchment

#0  0x00002b7f038a838d in poll () from /lib64/libc.so.6
#1  0x00002b7f085c1729 in Curl_poll () from /lib64/libcurl.so.4
#2  0x00002b7f085ba60d in curl_multi_wait () from /lib64/libcurl.so.4
#3  0x00002b7f085b368f in curl_easy_perform () from /lib64/libcurl.so.4
#4  0x00002b7f083851ff in slurm_jobcomp_set_location () from /usr/lib64/slurm/jobcomp_elasticsearch.so
#5  0x00002b7f0309b90f in g_slurm_jobcomp_init () from /usr/lib64/slurm/libslurmfull.so
#6  0x000000000049b671 in read_slurm_conf ()
#7  0x000000000042bd62 in main ()

None timeout in plugin jobcomp/elasticsearch

$ curl -vvv http://@server-http/elasticsearch7

* About to connect() to @server-http port 80 (#0)
*   Trying @ip
* Connected to @server-http @ip port 80 (#0)
> GET /elasticsearch7 HTTP/1.1
> User-Agent: curl/7.29.0
> Host: @server-http 
> Accept: */*
>
Comment 9 Regine Gaudin 2020-01-23 03:13:40 MST 
Created attachment 12811 [details]
slurm-18.08.6-elasticsearch.patch

Here is a better formatted version of the previous patch we are using.
Comment 10 Alejandro Sanchez 2020-01-27 08:24:00 MST 
*** Ticket 8379 has been marked as a duplicate of this ticket. ***
Comment 12 Nate Rini 2020-01-27 10:29:52 MST 
(In reply to Regine Gaudin from comment #9)
> Created attachment 12811 [details]
> slurm-18.08.6-elasticsearch.patch
> 
> Here is a better formatted version of the previous patch we are using.

Thanks for the update, we are currently looking at merging a modified version of this patch.
Comment 16 Nate Rini 2020-02-01 13:10:15 MST 
Regine,

After some consideration, we decided to implement the request via JobCompParams (per bug#6673 comment#1 point#6). The patch is now upstream for 20.02:

> https://github.com/SchedMD/slurm/commit/a4e0c97f74eef8fb8e0fc985d44918094c6e49cb

Please respond if you have any questions or issues.

Thanks,
--Nate