Ticket 9358 - Need assistance getting GRES information into the prolog
Summary: Need assistance getting GRES information into the prolog
Status: RESOLVED DUPLICATE of ticket 9389
Alias: None
Product: Slurm
Classification: Unclassified
Component: Other (show other tickets)
Version: 20.02.3
Hardware: Linux Linux
: 4 - Minor Issue
Assignee: Dominik Bartkiewicz
QA Contact:
URL:
Depends on:
Blocks:
 
Reported: 2020-07-08 07:34 MDT by Trey Dockendorf
Modified: 2020-07-17 06:42 MDT (History)
2 users (show)

See Also:
Site: Ohio State OSC
Slinky Site: ---
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
Google sites: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
NoveTech Sites: ---
Nvidia HWinf-CS Sites: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Tzag Elita Sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed:
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this ticket.
Description Trey Dockendorf 2020-07-08 07:34:17 MDT 
At OSC we have a few GRESes that require the prolog to make changes on the system as root if the job requests the GRES. One such gres is the "ime" GRES which is for DDN's burst buffer.  We need to have a user say --gres=ime and then be able to detect that in the prolog and start the IME service on the compute node as well as stop the daemon at job in end the epilog.  We already force IME jobs to be exclusive to avoid starting/stopping the daemon when other jobs may be using the IME burst buffer.

Based on discussions during OSC's consulting with SchedMD it seems like we may need to implement a SPANK plugin or PrEP plugin.
Comment 3 Dominik Bartkiewicz 2020-07-10 10:48:17 MDT 
Hi

Currently, you don’t have direct (without sending requests to slurmctld) access to gres info on spank or prolog script level.

This info is available for PrEp plugin.
In 20.02 prolog is called from "prep/script" plugin,
this allows forwarding gres info to prolog script as environment variables.

Another solution can be the creation of its PrEp plugin.

Let me know if you are interested in passing to prolog some new environments, or you need some assistance on developing PrEp plugin.

Dominik
Comment 4 Trey Dockendorf 2020-07-10 11:15:49 MDT 
Sending environment variables to the prolog sounds simpler, though I am not sure how that would be done.  How would passing new environment variables be done?  Any help on that aspect would be appreciated.
Comment 6 Trey Dockendorf 2020-07-17 06:42:49 MDT 
This was solved in another case: https://bugs.schedmd.com/show_bug.cgi?id=9389

I am using CLI filter plugin and job submit plugin to set SLURM_JOB_GRES environment variable and then reading that environment variable in SPANK plugin to set SPANK_GRES which is accessible in prolog.

The C plugins are here: https://github.com/treydock/slurm/tree/gres_env/src/plugins/cli_filter/gres_env

I am currently using Lua cli_filter plugin to replace the C cli_filter/gres_env plugin but kept the C approach in case we run into issues with Lua approach.

*** This ticket has been marked as a duplicate of ticket 9389 ***