15425 – Job Submit from REST API doesn't create User Environment Variables

Ticket 15425 - Job Submit from REST API doesn't create User Environment Variables

Summary: Job Submit from REST API doesn't create User Environment Variables

Status:	RESOLVED TIMEDOUT

Alias:	None

Product:	Slurm
Classification:	Unclassified
Component:	slurmrestd (show other tickets)
Version:	21.08.8
Hardware:	Linux Linux

Severity:	4 - Minor Issue
Assignee:	Nate Rini
QA Contact:

URL:

Depends on:
Blocks:

Reported:	2022-11-15 13:33 MST by Tom Ryan
Modified:	2023-02-22 11:24 MST (History)
CC List:	0 users

See Also:
Site:	Corning
Alineos Sites:	---
Atos/Eviden Sites:	---
Confidential Site:	---
Coreweave sites:	---
Cray Sites:	---
DS9 clusters:	---
HPCnow Sites:	---
HPE Sites:	---
IBM Sites:	---
NOAA SIte:	---
OCF Sites:	---
Recursion Pharma Sites:	---
SFW Sites:	---
SNIC sites:	---
Linux Distro:	---
Machine Name:
CLE Version:
Version Fixed:
Target Release:	---
DevPrio:	---
Emory-Cloud Sites:	---

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this ticket.

Description Tom Ryan 2022-11-15 13:33:02 MST

I'm attempting to submit a Slurm job using the api to create a session on our HPC cluster and start up TurboVNC. When using the API critical environment variables are missing for TurboVNC and GNOME to start up. I'm able to submit the job using sbatch but not the restapi. It appears the sbatch inherits environment variables from the logged in user and the restapi doesn't. Here is output from a sample job submitted just running printenv

LD_LIBRARY_PATH=/cm/shared/apps/slurm/current/lib64/slurm:/cm/shared/apps/slurm/current/lib64
SLURM_NODEID=0
SLURM_TASK_PID=2210891
HOSTNAME=focus01
ENVIRONMENT=BATCH
SLURM_PROCID=0
SLURM_JOB_GID=2577
SLURMD_NODENAME=focus01
SLURM_TASKS_PER_NODE=4
SLURM_NNODES=1
SLURM_GET_USER_ENV=1
PWD=/home/ryantp
SLURM_JOB_NODELIST=focus01
SLURM_CLUSTER_NAME=slurm
SLURM_NODELIST=focus01
SLURM_NTASKS=4
SLURM_JOB_CPUS_PER_NODE=4
SLURM_TOPOLOGY_ADDR=focus01
SLURM_WORKING_CLUSTER=slurm:mgmt01:6817:9472:109
SLURM_JOB_NAME=viz-test
TMPDIR=/tmp
SLURM_JOBID=11432
SLURM_CONF=/cm/shared/apps/slurm/var/etc/slurm/slurm.conf
SLURM_NODE_ALIASES=(null)
SLURM_JOB_QOS=normal
SLURM_TOPOLOGY_ADDR_PATTERN=node
SLURM_CPUS_ON_NODE=4
SLURM_JOB_NUM_NODES=1
SLURM_MEM_PER_NODE=40960
SLURM_JOB_UID=1830
SLURM_JOB_PARTITION=sc_focus_gpu
SLURM_JOB_USER=ryantp
SLURM_NPROCS=4
SHLVL=1
SLURM_JOB_ACCOUNT=sc_users
SLURM_GTIDS=0
PATH=/cm/shared/apps/slurm/current/sbin:/cm/shared/apps/slurm/current/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/ryantp/bin:/opt/TurboVNC/bin/:/opt/VirtualGL/bin/
SLURM_JOB_ID=11432
SLURM_LOCALID=0
_=/usr/bin/printenv

Comment 3 Nate Rini 2022-11-22 20:04:57 MST

(In reply to Tom Ryan from comment #0)
> I'm attempting to submit a Slurm job using the api to create a session on
> our HPC cluster and start up TurboVNC. When using the API critical
> environment variables are missing for TurboVNC and GNOME to start up. I'm
> able to submit the job using sbatch but not the restapi. It appears the
> sbatch inherits environment variables from the logged in user and the
> restapi doesn't. Here is output from a sample job submitted just running
> printenv

Yes, slurmrestd submitted jobs do not inherit any environment variables. They must be provided explicitly via the environment list.

Please see the environment field under V0.0.38_JOB_PROPERTIES:
> https://slurm.schedmd.com/rest_api.html#v0.0.38_job_properties

Comment 4 Nate Rini 2022-12-05 13:31:55 MST

Looks like there are no more questions. Please respond to this ticket with any more related questions.

Comment 5 Tom Ryan 2022-12-05 13:36:35 MST

I have no further questions. My preference is to have an environment created when the slurm job is run through the API as opposed to sending all environment variables. This makes it difficult to send a job on behalf of a user.

-Tom

From: bugs@schedmd.com <bugs@schedmd.com>
Sent: Monday, December 5, 2022 3:32 PM
To: Ryan, Thomas P <ryantp@corning.com>
Subject: [EXTERNAL]--[Bug 15425] Job Submit from REST API doesn't create User Environment Variables

Nate Rini<mailto:nate@schedmd.com> changed bug 15425<https://bugs.schedmd.com/show_bug.cgi?id=15425>
What
Removed
Added
Status
OPEN
RESOLVED
Resolution
---
INFOGIVEN
Comment # 4<https://bugs.schedmd.com/show_bug.cgi?id=15425#c4> on bug 15425<https://bugs.schedmd.com/show_bug.cgi?id=15425> from Nate Rini<mailto:nate@schedmd.com>

Looks like there are no more questions. Please respond to this ticket with any

more related questions.

________________________________
You are receiving this mail because:

  *   You reported the bug.

Comment 6 Nate Rini 2022-12-05 13:40:20 MST

(In reply to Tom Ryan from comment #5)
> I have no further questions. My preference is to have an environment created
> when the slurm job is run through the API as opposed to sending all
> environment variables. This makes it difficult to send a job on behalf of a
> user.

Can you please clarify where this environment should be coming from? It is generally expected slurmrestd will be started by systemd and won't have any (useful) environment to pass along.

Comment 7 Tom Ryan 2022-12-05 13:54:05 MST

I have a web application that is submitting a job on a user’s behalf to slurmrestd, the job will start TurboVNC for the user. TurboVNC cannot start because it’s missing environment variables such as $USER, $HOME, and more (I haven’t tracked down all of the variables yet)

Currently I’m submitting the jobs for the user using sudo -u <username> sbatch in order to carry over all the users environment variables in order to start the TurboVNC session. I’d like a cleaner way to submit the jobs. I’ve attempted to source files in order to create the environment for the users as part of the job submission script.

-Tom

From: bugs@schedmd.com <bugs@schedmd.com>
Sent: Monday, December 5, 2022 3:40 PM
To: Ryan, Thomas P <ryantp@corning.com>
Subject: [EXTERNAL]--[Bug 15425] Job Submit from REST API doesn't create User Environment Variables

Nate Rini<mailto:nate@schedmd.com> changed bug 15425<https://bugs.schedmd.com/show_bug.cgi?id=15425>
What
Removed
Added
Status
RESOLVED
OPEN
Resolution
INFOGIVEN
---
Comment # 6<https://bugs.schedmd.com/show_bug.cgi?id=15425#c6> on bug 15425<https://bugs.schedmd.com/show_bug.cgi?id=15425> from Nate Rini<mailto:nate@schedmd.com>

(In reply to Tom Ryan from comment #5<show_bug.cgi?id=15425#c5>)

> I have no further questions. My preference is to have an environment created

> when the slurm job is run through the API as opposed to sending all

> environment variables. This makes it difficult to send a job on behalf of a

> user.

Can you please clarify where this environment should be coming from? It is

generally expected slurmrestd will be started by systemd and won't have any

(useful) environment to pass along.

________________________________
You are receiving this mail because:

  *   You reported the bug.

Comment 8 Tom Ryan 2022-12-05 14:32:50 MST

The lack of XDG_environment variables also seem to prevent certain applications from starting up or from functioning correctly.

I’d like the slurmrest api to treat jobs submitted similar to the way sbatch or srun work in regards to creating all the variables a user might get in their environment when they login or submit a job.

From: bugs@schedmd.com <bugs@schedmd.com>
Sent: Monday, December 5, 2022 3:40 PM
To: Ryan, Thomas P <ryantp@corning.com>
Subject: [EXTERNAL]--[Bug 15425] Job Submit from REST API doesn't create User Environment Variables

Nate Rini<mailto:nate@schedmd.com> changed bug 15425<https://bugs.schedmd.com/show_bug.cgi?id=15425>
What
Removed
Added
Status
RESOLVED
OPEN
Resolution
INFOGIVEN
---
Comment # 6<https://bugs.schedmd.com/show_bug.cgi?id=15425#c6> on bug 15425<https://bugs.schedmd.com/show_bug.cgi?id=15425> from Nate Rini<mailto:nate@schedmd.com>

(In reply to Tom Ryan from comment #5<show_bug.cgi?id=15425#c5>)

> I have no further questions. My preference is to have an environment created

> when the slurm job is run through the API as opposed to sending all

> environment variables. This makes it difficult to send a job on behalf of a

> user.

Can you please clarify where this environment should be coming from? It is

generally expected slurmrestd will be started by systemd and won't have any

(useful) environment to pass along.

________________________________
You are receiving this mail because:

  *   You reported the bug.

Comment 9 Nate Rini 2022-12-08 13:36:41 MST

(In reply to Tom Ryan from comment #7)
> I have a web application that is submitting a job on a user’s behalf to
> slurmrestd, the job will start TurboVNC for the user. TurboVNC cannot start
> because it’s missing environment variables such as $USER, $HOME, and more (I
> haven’t tracked down all of the variables yet)
> 
> Currently I’m submitting the jobs for the user using sudo -u <username>
> sbatch in order to carry over all the users environment variables in order
> to start the TurboVNC session. I’d like a cleaner way to submit the jobs.
> I’ve attempted to source files in order to create the environment for the
> users as part of the job submission script.

(In reply to Tom Ryan from comment #8)
> The lack of XDG_environment variables also seem to prevent certain
> applications from starting up or from functioning correctly.
> 
> I’d like the slurmrest api to treat jobs submitted similar to the way sbatch
> or srun work in regards to creating all the variables a user might get in
> their environment when they login or submit a job.

Looking into your request.

Comment 10 Nate Rini 2022-12-08 13:40:26 MST

(In reply to Nate Rini from comment #9)
> (In reply to Tom Ryan from comment #7)
> > I have a web application that is submitting a job on a user’s behalf to
> > slurmrestd, the job will start TurboVNC for the user. TurboVNC cannot start
> > because it’s missing environment variables such as $USER, $HOME, and more (I
> > haven’t tracked down all of the variables yet)
> > 
> > Currently I’m submitting the jobs for the user using sudo -u <username>
> > sbatch in order to carry over all the users environment variables in order
> > to start the TurboVNC session. I’d like a cleaner way to submit the jobs.
> > I’ve attempted to source files in order to create the environment for the
> > users as part of the job submission script.
> 
> (In reply to Tom Ryan from comment #8)
> > The lack of XDG_environment variables also seem to prevent certain
> > applications from starting up or from functioning correctly.
> > 
> > I’d like the slurmrest api to treat jobs submitted similar to the way sbatch
> > or srun work in regards to creating all the variables a user might get in
> > their environment when they login or submit a job.
> 
> Looking into your request.

Please verify if the jobs have the "get_user_environment" field set to true at submission time.
> https://slurm.schedmd.com/rest_api.html#v0.0.38_job_properties

Comment 11 Nate Rini 2023-01-04 13:57:16 MST

Any updates?

Comment 12 Tom Ryan 2023-01-04 14:57:02 MST

Sorry yes,

I was able to submit the job and start up a TurboVNC session. I had to define the environment variables for USER HOME PATH TEMP. Once the job was submitted I had to define more variables before TurboVNC would start up a GNOME Desktop session, see below. But all is working now

    export XDG_RUNTIME_DIR=$TEMP
    export XDG_SESSION_ID=$(</proc/self/sessionid)
    export XDG_SESSION_TYPE=tty
    export XDG_SESSION_CLASS=user

-Tom

From: bugs@schedmd.com <bugs@schedmd.com>
Sent: Wednesday, January 4, 2023 3:57 PM
To: Ryan, Thomas P <ryantp@corning.com>
Subject: [EXTERNAL]--[Bug 15425] Job Submit from REST API doesn't create User Environment Variables


Caution: This email originated from outside of Corning. Do not click links or open attachments unless you recognize the sender and know the content is safe. Report phishing by using the “PhishMe Reporter Button” above or forward the email to Phishyemails@corning.com<mailto:Phishyemails@corning.com>.
Comment # 11<https://bugs.schedmd.com/show_bug.cgi?id=15425#c11> on bug 15425<https://bugs.schedmd.com/show_bug.cgi?id=15425> from Nate Rini<mailto:nate@schedmd.com>

Any updates?

________________________________
You are receiving this mail because:

  *   You reported the bug.

Comment 13 Nate Rini 2023-01-09 08:12:41 MST

(In reply to Tom Ryan from comment #12)
> I was able to submit the job and start up a TurboVNC session.

Was this job started with get_user_environment=true?

Comment 14 Tom Ryan 2023-01-09 08:25:42 MST

It was, however the “get_user_environment” doesn’t appear to add any additional variables. I’ve attempted to submit the job with and without “get_user_environment”, there isn’t a difference.

The jobs are submitted on the users behalf using a web application so there isn’t an environment to carry over. It sounds like it instead would need to “be created” similar to an ssh login, or login with gdm, when all those variables are set.

-Tom

From: bugs@schedmd.com <bugs@schedmd.com>
Sent: Monday, January 9, 2023 10:13 AM
To: Ryan, Thomas P <ryantp@corning.com>
Subject: [EXTERNAL]--[Bug 15425] Job Submit from REST API doesn't create User Environment Variables

Caution: This email originated from outside of Corning. Do not click links or open attachments unless you recognize the sender and know the content is safe. Report phishing by using the “PhishMe Reporter Button” above or forward the email to Phishyemails@corning.com<mailto:Phishyemails@corning.com>.
Comment # 13<https://bugs.schedmd.com/show_bug.cgi?id=15425#c13> on bug 15425<https://bugs.schedmd.com/show_bug.cgi?id=15425> from Nate Rini<mailto:nate@schedmd.com>

(In reply to Tom Ryan from comment #12<show_bug.cgi?id=15425#c12>)

> I was able to submit the job and start up a TurboVNC session.

Was this job started with get_user_environment=true?

________________________________
You are receiving this mail because:

  *   You reported the bug.

Comment 16 Nate Rini 2023-01-09 09:09:57 MST

(In reply to Tom Ryan from comment #14)
> It was, however the “get_user_environment” doesn’t appear to add any
> additional variables. I’ve attempted to submit the job with and without
> “get_user_environment”, there isn’t a difference.
> 
> The jobs are submitted on the users behalf using a web application so there
> isn’t an environment to carry over. It sounds like it instead would need to
> “be created” similar to an ssh login, or login with gdm, when all those
> variables are set.

The variables listed in comment#12 are set via pam_systemd which is disabled for Slurm due it causing systemd to steal processes from Slurm's control. Full details are in bug#5920.

We have the workaround for the conflict documented here to get the similiar functionality:
> https://slurm.schedmd.com/pam_slurm_adopt.html#PAM_CONFIG

Applying the changes there along with using get_user_environment=true should fix the issue. Just setting the TaskProlog from the link above should resolve the environment variable issue.

Comment 17 Nate Rini 2023-01-16 16:40:06 MST

(In reply to Nate Rini from comment #16)

Have these instructions helped?

Comment 18 Nate Rini 2023-02-22 11:24:11 MST

(In reply to Nate Rini from comment #17)
> (In reply to Nate Rini from comment #16)
> 
> Have these instructions helped?

I'm going to time this ticket out. Please respond at your convenience and we can continue debugging.