Hi Regine - I have moved this issue over to an enhancement. Right not we do not have plans to work on this however now that this issue is logged it will be something we consider in the future when making changes and update to the elasticsearch code.

Hi Regine,

The plugin already creates a joinable thread to handle the job indexation work. Given a list of finished and yet not indexed jobs, the thread continuously attempts to index each job, with a separate retry interval of 30s. If during this process the elastic server is down, then it logs a message specifying that it can't connect to the server. In my testing, this situation isn't leaving the ctld stuck in an infinite wait.

Do you have any reproducer for what you are experiencing?

Here is the stack of the slurmctld thread which was stuck: (as no timeout was set). It might be during a restarting or reconfigure...


#0  0x00002b7f038a838d in poll () from /lib64/libc.so.6
#1  0x00002b7f085c1729 in Curl_poll () from /lib64/libcurl.so.4
#2  0x00002b7f085ba60d in curl_multi_wait       CURLOPT_TIMEOUT
              Pass  a  long as parameter containing the maximum time in seconds that you allow the libcurl transfer
              operation to take. Normally, name lookups can take a considerable time  and  limiting  operations  to
              less than a few minutes risk aborting perfectly normal operations. This option will cause curl to use
              the SIGALRM to enable time-outing system calls.

              In unix-like systems, this might cause signals to be used unless CURLOPT_NOSIGNAL is set.

              Default timeout is 0 (zero) which means it never times out.
() from /lib64/libcurl.so.4
#3  0x00002b7f085b368f in curl_easy_perform () from /lib64/libcurl.so.4
#4  0x00002b7f083851ff in slurm_jobcomp_set_location () from /usr/lib64/slurm/jobcomp_elasticsearch.so
#5  0x00002b7f0309b90f in g_slurm_jobcomp_init () from /usr/lib64/slurm/libslurmfull.so
#6  0x000000000049b671 in read_slurm_conf ()
#7  0x000000000042bd62 in main ()

While http server was not responding

$ curl -vvv http://http-server/elasticsearch7

* About to connect() to http-server port 80 (#0)
*   Trying @ip.
* Connected to http-servr (@ip) port 80 (#0)
> GET /elasticsearch7 HTTP/1.1
> User-Agent: curl/7.29.0
> Host: http-server
> Accept: */*
>
=> stuck

Hey Regine,

thanks for your feedback. I've managed to reproduce this, but I am gonna mark this as a duplicate of bug 8325. The symptoms there are the same, just in the other bug you propose a contributed solution that we are already leveraging internally.

*** This ticket has been marked as a duplicate of ticket 8325 ***

I've opened the bug 8325 for allowing curl timeout setting in slurm.

I've opened this one for the following reason:
even with a curl_timeout set, if the http server is completely down, couldn't we fall in controller thread balancing between stuck "read_slurm_conf" during curl_timeout then running then again stuck... ?

(In reply to Regine Gaudin from comment #7)
> I've opened the bug 8325 for allowing curl timeout setting in slurm.
> 
> I've opened this one for the following reason:
> even with a curl_timeout set, if the http server is completely down,
> couldn't we fall in controller thread balancing between stuck
> "read_slurm_conf" during curl_timeout then running then again stuck... ?

The curl_easy_perform is a blocking call that exacerbates the problem when it's called in the plugin set_location() function from read_slurm_conf as you indicate. When it's called from the plugin separate thread, it at least doesn't leave the controller stuck in the meantime.

In the original bug, we're already leveraging what to do with that curl_easy_perform call in the set_location(). Originally it was intended to log an early error if a connection test to the just parsed location failed. But the call in there is not strictly necessary since the separate thread will attempt to connect later for each finished job and inform if there's any problem with the indexation. Removing the curl logic entirely from slurm_jobcomp_set_location() is an option. Putting the logic into a separate thread is another. Using the non-blocking approach curl_multi_perform() as the libcurl documentation suggests would be another one.

At the end of the day, the motivation for the timeout artifact comes from this problem, and it is being discussed internally there. So I am inclined to reclose this as a duplicate of that one. We will update you there once we come up with a solution.