Ticket 6673

Summary: Elasticsearch plugin - use password file
Product: Slurm Reporter: Aaron Holt <aaron.holt>
Component: OtherAssignee: Alejandro Sanchez <asanchezg>
Status: RESOLVED INFOGIVEN QA Contact:
Severity: 4 - Minor Issue    
Priority: --- CC: asanchezg, jbooth, nrini
Version: 17.11.7   
Hardware: Linux   
OS: Linux   
See Also: https://bugs.schedmd.com/show_bug.cgi?id=6109
https://bugs.schedmd.com/show_bug.cgi?id=8333
https://bugs.schedmd.com/show_bug.cgi?id=8325
Site: University of Colorado Slinky Site: ---
Alineos Sites: --- Atos/Eviden Sites: ---
Confidential Site: --- Coreweave sites: ---
Cray Sites: --- DS9 clusters: ---
Google sites: --- HPCnow Sites: ---
HPE Sites: --- IBM Sites: ---
NOAA SIte: --- NoveTech Sites: ---
Nvidia HWinf-CS Sites: --- OCF Sites: ---
Recursion Pharma Sites: --- SFW Sites: ---
SNIC sites: --- Tzag Elita Sites: ---
Linux Distro: --- Machine Name:
CLE Version: Version Fixed:
Target Release: --- DevPrio: ---
Emory-Cloud Sites: ---

Description Aaron Holt 2019-03-11 13:28:52 MDT 
Hi,

Is there a way to use a password file for the Elasticsearch jobcomp plugin? Currently, our slurm.conf is readable by all users. If I put JobCompPass and JobCompUser in slurm.conf, anyone who looks will have our Elasticsearch credentials. Other workarounds would be fine, too.

Thanks!
-Aaron
Comment 1 Alejandro Sanchez 2019-03-11 14:26:15 MDT 
Hi,

JobComp[User|Pass] options don't currently apply to this plugin. As documented in guide[1], the only available options that apply are JobCompType, JobCompLoc and DebugFlags.

Since long ago I have a list of nice things to enhance the functionality offered by the plugin. Unfortunately, absent someone willing to sponsor such changes, or a contributed patch providing expansions, SchedMD cannot say when/if this will ever be addressed. Some of the ideas that I have in mind include:

1. Let the user decide the index name and index type instead of forcing the current fixed names /slurm/jobcomp.

2. Use the json-c library instead of manually generating a JSON formatted string for each finished job.

3. Add support for https://www.elastic.co/products/x-pack X-Pack credentials and potentially take advantage of the already available JobComp[User|Pass].

4. Parse the REST response by using the https://curl.haxx.se/libcurl/c/curl_easy_getinfo.html libcurl method. This will be a safer way to parse the different response parts (CURLINFO_RESPONSE_CODE, ...) than manually using strtok.

5. Move the libcurl C API setup to a common function under src/common. Potential new code using libcurl could take advantage of that (i.e. InfluxDB).

6. Add a new JobCompParams slurm.conf option to specify different params such as ESExpandHostlist (which indicates whether to store the node list in a contracted or expanded format) or ESIndex=[none|day|week|month|year] to append a different date to the index name at different frequencies.

[1] https://slurm.schedmd.com/elasticsearch.html