Ticket 7588

Summary: Is there a way to set an env variable via srun by default?
Product: Slurm Reporter: darrellp
Component: OtherAssignee: Gavin D. Howard <gavin>
Status: RESOLVED INFOGIVEN QA Contact:
Severity: 4 - Minor Issue    
Priority: ---    
Version: 19.05.0   
Hardware: Linux   
OS: Linux   
Site: Allen AI Slinky Site: ---
Alineos Sites: --- Atos/Eviden Sites: ---
Confidential Site: --- Coreweave sites: ---
Cray Sites: --- DS9 clusters: ---
Google sites: --- HPCnow Sites: ---
HPE Sites: --- IBM Sites: ---
NOAA SIte: --- NoveTech Sites: ---
Nvidia HWinf-CS Sites: --- OCF Sites: ---
Recursion Pharma Sites: --- SFW Sites: ---
SNIC sites: --- Tzag Elita Sites: ---
Linux Distro: --- Machine Name:
CLE Version: Version Fixed:
Target Release: --- DevPrio: ---
Emory-Cloud Sites: ---
Attachments: Slurm.conf

Description darrellp 2019-08-16 12:58:44 MDT 
Created attachment 11270 [details]
Slurm.conf

Minor information request

This will be a little bit of a strange one, so bear with us. We would like to ask is there a way to set a default env variable for srun sessions one the server side. 

The use case is that we have had creative users figure out if they run this command, they get launch a bash shell on a machine and get total access to all the resources, outside of slurm control:

srun -w allennlp-server3  bash -i

Now we are going to address this socially, but we were wondering if there was a techincal way to lock this down. We found that if this env var is set, the user cannot see the GPUs

export CUDA_VISIBLE_DEVICES=

Is there anyway to have this be the default, when the user calls srun? 

Thanks
D
Comment 1 Gavin D. Howard 2019-08-16 15:43:52 MDT 
Setting that environment variable might work, for now. However, Slurm has a much better solution for preventing users from accessing resources outside of Slurm: the task/cgroup plugin.

tl;dr: The task/cgroup plugin makes all resources available in a cgroup when a job is submitted, so a job cannot use any resources that it wasn't given.

All of the relevant documentation you might need is at https://slurm.schedmd.com/cgroups.html and at https://slurm.schedmd.com/cgroup.conf.html .

Of course, the first thing you need to do is change this line in your `slurm.conf`:

> TaskPlugin=task/affinity

to

> TaskPlugin=task/cgroup

Please let me know if you have any questions.
Comment 3 Gavin D. Howard 2019-08-23 10:50:27 MDT 
Because I haven't heard back from you in a while, I am going to mark this bug as RESOLVED - INFOGIVEN for now.

Do feel free to reopen if you need more help.
Comment 4 darrellp 2019-08-23 11:03:00 MDT 
Sorry for not replying Gavin, we appreciate the reply, we were following
the cgroup rabbit hole, but we feel we have a handle on this for the time
being. Thanks for your help

D

On Fri, Aug 23, 2019 at 9:50 AM <bugs@schedmd.com> wrote:

> Gavin D. Howard <gavin@schedmd.com> changed bug 7588
> <https://bugs.schedmd.com/show_bug.cgi?id=7588>
> What Removed Added
> Resolution --- INFOGIVEN
> Status OPEN RESOLVED
>
> *Comment # 3 <https://bugs.schedmd.com/show_bug.cgi?id=7588#c3> on bug
> 7588 <https://bugs.schedmd.com/show_bug.cgi?id=7588> from Gavin D. Howard
> <gavin@schedmd.com> *
>
> Because I haven't heard back from you in a while, I am going to mark this bug
> as RESOLVED - INFOGIVEN for now.
>
> Do feel free to reopen if you need more help.
>
> ------------------------------
> You are receiving this mail because:
>
>    - You reported the bug.
>
>