Ticket 9809 - Add environment variables effective in user's job script depending on CPU type
Summary: Add environment variables effective in user's job script depending on CPU type
Status: RESOLVED INFOGIVEN
Alias: None
Product: Slurm
Classification: Unclassified
Component: Other (show other tickets)
Version: 20.02.3
Hardware: Linux Linux
: 4 - Minor Issue
Assignee: Marcin Stolarek
QA Contact:
URL:
Depends on:
Blocks:
 
Reported: 2020-09-11 23:55 MDT by issp2020support
Modified: 2020-10-20 07:07 MDT (History)
2 users (show)

See Also:
Site: U of Tokyo
Slinky Site: ---
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
Google sites: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
NoveTech Sites: ---
Nvidia HWinf-CS Sites: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Tzag Elita Sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed:
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this ticket.
Description issp2020support 2020-09-11 23:55:59 MDT 
I (administrator) would like to add environment variables effective in user's job script.

The system has two CPU types (Intel, AMD), and the following environment variables are only required for AMD compute nodes.
   export MKL_DEBUG_CPU_TYPE=5
   export MKL_ENABLE_INSTRUCTIONS=AVX2

I tried setting the following shell script to Prolog, SrunProlog and TaskProlog, then none of them added the variables.

Could you tell me what to do?

###################################
#!/bin/sh

amd=`grep vendor_id /proc/cpuinfo | uniq | grep AMD | wc -l`;

if [ $amd -eq 1 ]; then
   export MKL_DEBUG_CPU_TYPE=5
   export MKL_ENABLE_INSTRUCTIONS=AVX2
fi
###################################
Comment 1 Marcin Stolarek 2020-09-14 05:39:08 MDT 
You can set the variable from TaskProlog, but you need to use syntax like:
>echo 'export MKL_ENABLE_INSTRUCTIONS=AVX2'

Technically user job is forked separately from prolog, script environment is not "inherited" by the job. However, the standard output of the script is interpreted by Slurm, and if it contains "export ..." line than the variable is set during execv call to job script.[1]

Let me know if you have further questions on that. 

cheers,
Marcin
[1]https://slurm.schedmd.com/prolog_epilog.html
Comment 2 issp2020support 2020-09-14 12:47:38 MDT 
Hi,

Thank you!
Now I can get expected results as follows.
Please close this bug.

$ sbatch -p amd --wrap "hostname; env | grep MKL"
Submitted batch job 5169
$ cat slurm-5169.out
amd-node1
MKL_DEBUG_CPU_TYPE=5
MKL_ENABLE_INSTRUCTIONS=AVX2

$ sbatch -p intel --wrap "hostname; env | grep MKL"
Submitted batch job 5170
$ cat slurm-5170.out
intel-node1
Comment 4 Marco Induni 2020-10-20 07:00:12 MDT 
(In reply to Marcin Stolarek from comment #1)
> You can set the variable from TaskProlog, but you need to use syntax like:
> >echo 'export MKL_ENABLE_INSTRUCTIONS=AVX2'
> 
> Technically user job is forked separately from prolog, script environment is
> not "inherited" by the job. However, the standard output of the script is
> interpreted by Slurm, and if it contains "export ..." line than the variable
> is set during execv call to job script.[1]
> 
> Let me know if you have further questions on that. 
> 
> cheers,
> Marcin
> [1]https://slurm.schedmd.com/prolog_epilog.html

Hi Marcin,
I've more or less the same issue, but in my case I needed to unset a user environment variable only for a specific partition. 
This way is working (using TaskProlog)

if [[ ${SLURM_JOB_PARTITION} == xfer ]]; then
 echo 'unset LD_PRELOAD'
fi

I wonder if this would be feasible on the Prolog, or in another place, so in case there are jobs with many tasks, this is started just once (we have user that hav lots of task and I'm worried about scasle/performance issues)

Thank you
Marco Induni
Comment 5 Marcin Stolarek 2020-10-20 07:07:53 MDT 
Marco,

Since it's a different topic - not "how to add", but rather is it worth or possible to do that differently. Could you please open a separate ticket to get appropriate attention our side?

I'm closing back this one.

cheers,
Marcin