Ticket 9358

Summary:	Need assistance getting GRES information into the prolog
Product:	Slurm	Reporter:	Trey Dockendorf <tdockendorf>
Component:	Other	Assignee:	Dominik Bartkiewicz <bart>
Status:	RESOLVED DUPLICATE	QA Contact:
Severity:	4 - Minor Issue
Priority:	---	CC:	bart, nate
Version:	20.02.3
Hardware:	Linux
OS:	Linux
See Also:	https://bugs.schedmd.com/show_bug.cgi?id=9357
Site:	Ohio State OSC	Alineos Sites:	---
Atos/Eviden Sites:	---	Confidential Site:	---
Coreweave sites:	---	Cray Sites:	---
DS9 clusters:	---	HPCnow Sites:	---
HPE Sites:	---	IBM Sites:	---
NOAA SIte:	---	OCF Sites:	---
Recursion Pharma Sites:	---	SFW Sites:	---
SNIC sites:	---	Linux Distro:	---
Machine Name:		CLE Version:
Version Fixed:		Target Release:	---
DevPrio:	---	Emory-Cloud Sites:	---

Description Trey Dockendorf 2020-07-08 07:34:17 MDT

At OSC we have a few GRESes that require the prolog to make changes on the system as root if the job requests the GRES. One such gres is the "ime" GRES which is for DDN's burst buffer.  We need to have a user say --gres=ime and then be able to detect that in the prolog and start the IME service on the compute node as well as stop the daemon at job in end the epilog.  We already force IME jobs to be exclusive to avoid starting/stopping the daemon when other jobs may be using the IME burst buffer.

Based on discussions during OSC's consulting with SchedMD it seems like we may need to implement a SPANK plugin or PrEP plugin.

Comment 3 Dominik Bartkiewicz 2020-07-10 10:48:17 MDT

Hi

Currently, you don’t have direct (without sending requests to slurmctld) access to gres info on spank or prolog script level.

This info is available for PrEp plugin.
In 20.02 prolog is called from "prep/script" plugin,
this allows forwarding gres info to prolog script as environment variables.

Another solution can be the creation of its PrEp plugin.

Let me know if you are interested in passing to prolog some new environments, or you need some assistance on developing PrEp plugin.

Dominik

Comment 4 Trey Dockendorf 2020-07-10 11:15:49 MDT

Sending environment variables to the prolog sounds simpler, though I am not sure how that would be done.  How would passing new environment variables be done?  Any help on that aspect would be appreciated.

Comment 6 Trey Dockendorf 2020-07-17 06:42:49 MDT

This was solved in another case: https://bugs.schedmd.com/show_bug.cgi?id=9389

I am using CLI filter plugin and job submit plugin to set SLURM_JOB_GRES environment variable and then reading that environment variable in SPANK plugin to set SPANK_GRES which is accessible in prolog.

The C plugins are here: https://github.com/treydock/slurm/tree/gres_env/src/plugins/cli_filter/gres_env

I am currently using Lua cli_filter plugin to replace the C cli_filter/gres_env plugin but kept the C approach in case we run into issues with Lua approach.

*** This ticket has been marked as a duplicate of ticket 9389 ***