Ticket 9809

Summary:	Add environment variables effective in user's job script depending on CPU type
Product:	Slurm	Reporter:	issp2020support
Component:	Other	Assignee:	Marcin Stolarek <cinek>
Status:	RESOLVED INFOGIVEN	QA Contact:
Severity:	4 - Minor Issue
Priority:	---	CC:	cinek, marco.induni
Version:	20.02.3
Hardware:	Linux
OS:	Linux
Site:	U of Tokyo	Slinky Site:	---
Alineos Sites:	---	Atos/Eviden Sites:	---
Confidential Site:	---	Coreweave sites:	---
Cray Sites:	---	DS9 clusters:	---
Google sites:	---	HPCnow Sites:	---
HPE Sites:	---	IBM Sites:	---
NOAA SIte:	---	NoveTech Sites:	---
Nvidia HWinf-CS Sites:	---	OCF Sites:	---
Recursion Pharma Sites:	---	SFW Sites:	---
SNIC sites:	---	Tzag Elita Sites:	---
Linux Distro:	---	Machine Name:
CLE Version:		Version Fixed:
Target Release:	---	DevPrio:	---
Emory-Cloud Sites:	---

Description issp2020support 2020-09-11 23:55:59 MDT

I (administrator) would like to add environment variables effective in user's job script.

The system has two CPU types (Intel, AMD), and the following environment variables are only required for AMD compute nodes.
   export MKL_DEBUG_CPU_TYPE=5
   export MKL_ENABLE_INSTRUCTIONS=AVX2

I tried setting the following shell script to Prolog, SrunProlog and TaskProlog, then none of them added the variables.

Could you tell me what to do?

###################################
#!/bin/sh

amd=`grep vendor_id /proc/cpuinfo | uniq | grep AMD | wc -l`;

if [ $amd -eq 1 ]; then
   export MKL_DEBUG_CPU_TYPE=5
   export MKL_ENABLE_INSTRUCTIONS=AVX2
fi
###################################

Comment 1 Marcin Stolarek 2020-09-14 05:39:08 MDT

You can set the variable from TaskProlog, but you need to use syntax like:
>echo 'export MKL_ENABLE_INSTRUCTIONS=AVX2'

Technically user job is forked separately from prolog, script environment is not "inherited" by the job. However, the standard output of the script is interpreted by Slurm, and if it contains "export ..." line than the variable is set during execv call to job script.[1]

Let me know if you have further questions on that. 

cheers,
Marcin
[1]https://slurm.schedmd.com/prolog_epilog.html

Comment 2 issp2020support 2020-09-14 12:47:38 MDT

Hi,

Thank you!
Now I can get expected results as follows.
Please close this bug.

$ sbatch -p amd --wrap "hostname; env | grep MKL"
Submitted batch job 5169
$ cat slurm-5169.out
amd-node1
MKL_DEBUG_CPU_TYPE=5
MKL_ENABLE_INSTRUCTIONS=AVX2

$ sbatch -p intel --wrap "hostname; env | grep MKL"
Submitted batch job 5170
$ cat slurm-5170.out
intel-node1

Comment 4 Marco Induni 2020-10-20 07:00:12 MDT

(In reply to Marcin Stolarek from comment #1)
> You can set the variable from TaskProlog, but you need to use syntax like:
> >echo 'export MKL_ENABLE_INSTRUCTIONS=AVX2'
> 
> Technically user job is forked separately from prolog, script environment is
> not "inherited" by the job. However, the standard output of the script is
> interpreted by Slurm, and if it contains "export ..." line than the variable
> is set during execv call to job script.[1]
> 
> Let me know if you have further questions on that. 
> 
> cheers,
> Marcin
> [1]https://slurm.schedmd.com/prolog_epilog.html

Hi Marcin,
I've more or less the same issue, but in my case I needed to unset a user environment variable only for a specific partition. 
This way is working (using TaskProlog)

if [[ ${SLURM_JOB_PARTITION} == xfer ]]; then
 echo 'unset LD_PRELOAD'
fi

I wonder if this would be feasible on the Prolog, or in another place, so in case there are jobs with many tasks, this is started just once (we have user that hav lots of task and I'm worried about scasle/performance issues)

Thank you
Marco Induni

Comment 5 Marcin Stolarek 2020-10-20 07:07:53 MDT

Marco,

Since it's a different topic - not "how to add", but rather is it worth or possible to do that differently. Could you please open a separate ticket to get appropriate attention our side?

I'm closing back this one.

cheers,
Marcin