Elasticsearch has removed the concept of a document type. https://www.elastic.co/blog/moving-from-types-to-typeless-apis-in-elasticsearch-7-0 Using the jobcomp plugin with 6.x results in deprecation warnings. At some point, all APIs that accept types will be removed and the plugin will not work at all. POSTing to $JobCompLoc/slurm/jobcomp needs to change to $JobCompLoc/slurm/_doc. This is a breaking change for users that will likely require reindexing old data. To ease the transition, I suggest that the plugin accept a "fully qualified" $JobCompLoc that includes the index name and either a legacy type or "_doc". This would also allow users to support multiple Slurm clusters logging to the same Elastic Search server but to different indices. For 19.05 and 20.02, if a "fully qualified" $JobCompLoc is specified, use it directly For 19.05, if a "server only" $JobCompLoc is specified, append '/slurm/jobcomp' For 20.02, if a "server only" $JobCompLoc is specified, append '/slurm/_doc' We are testing such a patch now; we will upload it soon.
Matt - This looks like it will become a contribution soon. Please let me know if I am incorrect in that assumption. We do have some other changes that we would like to make with regards to Elasticsearch however those all need to be evaluated together along with what is being proposed here. I look forward to seeing your contribution.
Created attachment 12737 [details] Allow JobCompLoc to be full URL in jobcomp/elasticsearch
If this is accepted (or there is indication that it will be accepted) we can update the documentation as well.
Hey Matt, thanks for the contribution. I've been testing with 7.5.2 and the plugin is still able to index jobs. It looks like both _index and _type are created: ... "_index" : "slurm", "_type" : "jobcomp", ... Looking at the documentation link (thanks for the reference) it looks like in 8.0 it won't be accepted anymore. Letting the whole endpoint be configurable is something I already had in mind long ago. At this point so late in 20.02, we're more inclined to target this for 20.11 alongside a bigger refactor I've been planning for some time now.
Matt, Given that elastic 8.x is not yet released and Slurm 20.11 is likely be released around November this year, our current plan is the following: For 20.02 document that the plugin appends "/slurm/jobcomp" index and mapping to the URL and that can't be changed, meaning that this version of Slurm won't work with elastic versions >= 8.x due to the elastic type deprecation. For 20.11 the hardcoded index and mapping concatenation will be removed, leaving the competence of configuring the full endpoint (including the port) to the admin through JobCompLoc. The URL won't be attempted to be sanitized by the plugin but instead let the libcurl library connection calls attempt to connect against the configured value and complain if malformed, not reachable or any other problem. Thus we don't plan to look for the scheme/protocol component of the configured value, skip it or the like as done with the contributed patch. The deprecation message logged to elasticsearch_deprecation.log looks like this: [2020-09-11T17:49:26,899][WARN ][o.e.d.r.a.d.RestDeleteAction] [polaris] [types removal] Specifying types in document index requests is deprecated, use the typeless endpoints instead (/{index}/_doc/{id}, /{index}/_doc, or /{index}/_create/{id}). So in 20.11 and up the admin will need to configure a valid typeless endpoint like i.e.: JobCompLoc=localhost:9200/slurm/_doc We've leveraged removing the hardcoded index and mapping concatenation in 20.02 and mention the change in the NEWS file. But we feel this would disrupt some of the current plugin users workflows plus we think main Linux distros current versions won't be shipping elastic 8.x by default, and so it looks safer to do the actual change in current master (future 20.11) with an appropriate entry in the RELEASE_NOTES file and just document a potential incompatiblity for 20.02 + elastic 8.x. With all this said, are you fine with this path forward? If so, I'll work on tackling it next week. Also, thanks again for reporting this issue and the contribution.
(In reply to Alejandro Sanchez from comment #6) > With all this said, are you fine with this path forward? If so, I'll work on > tackling it next week. Yes, this sounds fine to me. Thanks.
Hi Matt, The plan has been expressed with the following commits: 1 new commit pushed to slurm-20.02 b8b9f206 - Docs - add note about Elasticsearch 8.0 transition to typeless mode. 1 new commit pushed to master 55eb7bf8 - jobcomp/elasticsearch - make JobCompLoc fully configurable. I'm gonna go ahead and mark this as resolved. Thanks again for reporting and please feel free to reopen if there's any further question.
