Ticket 7588 - Is there a way to set an env variable via srun by default?
Summary: Is there a way to set an env variable via srun by default?
Status: RESOLVED INFOGIVEN
Alias: None
Product: Slurm
Classification: Unclassified
Component: Other (show other tickets)
Version: 19.05.0
Hardware: Linux Linux
: 4 - Minor Issue
Assignee: Gavin D. Howard
QA Contact:
URL:
Depends on:
Blocks:
 
Reported: 2019-08-16 12:58 MDT by darrellp
Modified: 2019-08-23 11:03 MDT (History)
0 users

See Also:
Site: Allen AI
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed:
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---

Attachments
Slurm.conf (1.11 KB, text/plain)
2019-08-16 12:58 MDT, darrellp 		Details
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this ticket.
Description darrellp 2019-08-16 12:58:44 MDT 
Created attachment 11270 [details]
Slurm.conf

Minor information request

This will be a little bit of a strange one, so bear with us. We would like to ask is there a way to set a default env variable for srun sessions one the server side. 

The use case is that we have had creative users figure out if they run this command, they get launch a bash shell on a machine and get total access to all the resources, outside of slurm control:

srun -w allennlp-server3  bash -i

Now we are going to address this socially, but we were wondering if there was a techincal way to lock this down. We found that if this env var is set, the user cannot see the GPUs

export CUDA_VISIBLE_DEVICES=

Is there anyway to have this be the default, when the user calls srun? 

Thanks
D
Comment 1 Gavin D. Howard 2019-08-16 15:43:52 MDT 
Setting that environment variable might work, for now. However, Slurm has a much better solution for preventing users from accessing resources outside of Slurm: the task/cgroup plugin.

tl;dr: The task/cgroup plugin makes all resources available in a cgroup when a job is submitted, so a job cannot use any resources that it wasn't given.

All of the relevant documentation you might need is at https://slurm.schedmd.com/cgroups.html and at https://slurm.schedmd.com/cgroup.conf.html .

Of course, the first thing you need to do is change this line in your `slurm.conf`:

> TaskPlugin=task/affinity

to

> TaskPlugin=task/cgroup

Please let me know if you have any questions.
Comment 3 Gavin D. Howard 2019-08-23 10:50:27 MDT 
Because I haven't heard back from you in a while, I am going to mark this bug as RESOLVED - INFOGIVEN for now.

Do feel free to reopen if you need more help.
Comment 4 darrellp 2019-08-23 11:03:00 MDT 
Sorry for not replying Gavin, we appreciate the reply, we were following
the cgroup rabbit hole, but we feel we have a handle on this for the time
being. Thanks for your help

D

On Fri, Aug 23, 2019 at 9:50 AM <bugs@schedmd.com> wrote:

> Gavin D. Howard <gavin@schedmd.com> changed bug 7588
> <https://bugs.schedmd.com/show_bug.cgi?id=7588>
> What Removed Added
> Resolution --- INFOGIVEN
> Status OPEN RESOLVED
>
> *Comment # 3 <https://bugs.schedmd.com/show_bug.cgi?id=7588#c3> on bug
> 7588 <https://bugs.schedmd.com/show_bug.cgi?id=7588> from Gavin D. Howard
> <gavin@schedmd.com> *
>
> Because I haven't heard back from you in a while, I am going to mark this bug
> as RESOLVED - INFOGIVEN for now.
>
> Do feel free to reopen if you need more help.
>
> ------------------------------
> You are receiving this mail because:
>
>    - You reported the bug.
>
>