Ticket 8325

Summary: Missing timeouts for elasticsearch plugin lead to stuck controller
Product: Slurm Reporter: Regine Gaudin <regine.gaudin>
Component: slurmctldAssignee: Nate Rini <nrini>
Status: RESOLVED FIXED QA Contact: Tim Wickberg <tim>
Severity: C - Contributions    
Priority: --- CC: asanchezg, tim
Version: 20.02.x   
Hardware: Linux   
OS: Linux   
See Also: https://bugs.schedmd.com/show_bug.cgi?id=8333
https://bugs.schedmd.com/show_bug.cgi?id=6673
https://bugs.schedmd.com/show_bug.cgi?id=13499
Site: CEA Slinky Site: ---
Alineos Sites: --- Atos/Eviden Sites: ---
Confidential Site: --- Coreweave sites: ---
Cray Sites: --- DS9 clusters: ---
Google sites: --- HPCnow Sites: ---
HPE Sites: --- IBM Sites: ---
NOAA SIte: --- NoveTech Sites: ---
Nvidia HWinf-CS Sites: --- OCF Sites: ---
Recursion Pharma Sites: --- SFW Sites: ---
SNIC sites: --- Tzag Elita Sites: ---
Linux Distro: --- Machine Name:
CLE Version: Version Fixed: 20.02
Target Release: --- DevPrio: ---
Emory-Cloud Sites: ---
Attachments: use two env variables for curl timeout
slurm-18.08.6-elasticsearch.patch

Description Regine Gaudin 2020-01-13 06:28:04 MST 
Created attachment 12715 [details]
use two env variables for curl timeout

We are using elasticsearch plugin with slurm.18. The http server did not respond after a oom killer and this problem leads to stuck slurmctld infinite wait.

Alghough we need to increase robustness and ha of http server, give a  possibility of setting timeouts in slurm.conf for plugin jobcomp/elasticsearch 
would prevent such occurance. Is it possible for schedmd to do it ?

As it means to add members to slurm_ctl_conf_t structure it is better if schedmd 
implements it. 

A "less clean" war consisting in adding environment variables read in jobcomp_elasticsearch.c and used in curl_set_opt(CURL_TIMEOUT) , curl_set_opt(CURL_CONNECTTIMEOUT) before curl_easy_perform have been used.
See in atatchment

#0  0x00002b7f038a838d in poll () from /lib64/libc.so.6
#1  0x00002b7f085c1729 in Curl_poll () from /lib64/libcurl.so.4
#2  0x00002b7f085ba60d in curl_multi_wait () from /lib64/libcurl.so.4
#3  0x00002b7f085b368f in curl_easy_perform () from /lib64/libcurl.so.4
#4  0x00002b7f083851ff in slurm_jobcomp_set_location () from /usr/lib64/slurm/jobcomp_elasticsearch.so
#5  0x00002b7f0309b90f in g_slurm_jobcomp_init () from /usr/lib64/slurm/libslurmfull.so
#6  0x000000000049b671 in read_slurm_conf ()
#7  0x000000000042bd62 in main ()

None timeout in plugin jobcomp/elasticsearch

$ curl -vvv http://@server-http/elasticsearch7

* About to connect() to @server-http port 80 (#0)
*   Trying @ip
* Connected to @server-http @ip port 80 (#0)
> GET /elasticsearch7 HTTP/1.1
> User-Agent: curl/7.29.0
> Host: @server-http 
> Accept: */*
>
Comment 9 Regine Gaudin 2020-01-23 03:13:40 MST 
Created attachment 12811 [details]
slurm-18.08.6-elasticsearch.patch

Here is a better formatted version of the previous patch we are using.
Comment 10 Alejandro Sanchez 2020-01-27 08:24:00 MST 
*** Ticket 8379 has been marked as a duplicate of this ticket. ***
Comment 12 Nate Rini 2020-01-27 10:29:52 MST 
(In reply to Regine Gaudin from comment #9)
> Created attachment 12811 [details]
> slurm-18.08.6-elasticsearch.patch
> 
> Here is a better formatted version of the previous patch we are using.

Thanks for the update, we are currently looking at merging a modified version of this patch.
Comment 16 Nate Rini 2020-02-01 13:10:15 MST 
Regine,

After some consideration, we decided to implement the request via JobCompParams (per bug#6673 comment#1 point#6). The patch is now upstream for 20.02:

> https://github.com/SchedMD/slurm/commit/a4e0c97f74eef8fb8e0fc985d44918094c6e49cb

Please respond if you have any questions or issues.

Thanks,
--Nate