Ticket 15425

(In reply to Tom Ryan from comment #0)
> I'm attempting to submit a Slurm job using the api to create a session on
> our HPC cluster and start up TurboVNC. When using the API critical
> environment variables are missing for TurboVNC and GNOME to start up. I'm
> able to submit the job using sbatch but not the restapi. It appears the
> sbatch inherits environment variables from the logged in user and the
> restapi doesn't. Here is output from a sample job submitted just running
> printenv

Yes, slurmrestd submitted jobs do not inherit any environment variables. They must be provided explicitly via the environment list.

Please see the environment field under V0.0.38_JOB_PROPERTIES:
> https://slurm.schedmd.com/rest_api.html#v0.0.38_job_properties

Looks like there are no more questions. Please respond to this ticket with any more related questions.

I have no further questions. My preference is to have an environment created when the slurm job is run through the API as opposed to sending all environment variables. This makes it difficult to send a job on behalf of a user.

-Tom

From: bugs@schedmd.com <bugs@schedmd.com>
Sent: Monday, December 5, 2022 3:32 PM
To: Ryan, Thomas P <ryantp@corning.com>
Subject: [EXTERNAL]--[Bug 15425] Job Submit from REST API doesn't create User Environment Variables

Nate Rini<mailto:nate@schedmd.com> changed bug 15425<https://bugs.schedmd.com/show_bug.cgi?id=15425>
What
Removed
Added
Status
OPEN
RESOLVED
Resolution
---
INFOGIVEN
Comment # 4<https://bugs.schedmd.com/show_bug.cgi?id=15425#c4> on bug 15425<https://bugs.schedmd.com/show_bug.cgi?id=15425> from Nate Rini<mailto:nate@schedmd.com>

Looks like there are no more questions. Please respond to this ticket with any

more related questions.

________________________________
You are receiving this mail because:

  *   You reported the bug.

(In reply to Tom Ryan from comment #5)
> I have no further questions. My preference is to have an environment created
> when the slurm job is run through the API as opposed to sending all
> environment variables. This makes it difficult to send a job on behalf of a
> user.

Can you please clarify where this environment should be coming from? It is generally expected slurmrestd will be started by systemd and won't have any (useful) environment to pass along.

I have a web application that is submitting a job on a user’s behalf to slurmrestd, the job will start TurboVNC for the user. TurboVNC cannot start because it’s missing environment variables such as $USER, $HOME, and more (I haven’t tracked down all of the variables yet)

Currently I’m submitting the jobs for the user using sudo -u <username> sbatch in order to carry over all the users environment variables in order to start the TurboVNC session. I’d like a cleaner way to submit the jobs. I’ve attempted to source files in order to create the environment for the users as part of the job submission script.


-Tom

From: bugs@schedmd.com <bugs@schedmd.com>
Sent: Monday, December 5, 2022 3:40 PM
To: Ryan, Thomas P <ryantp@corning.com>
Subject: [EXTERNAL]--[Bug 15425] Job Submit from REST API doesn't create User Environment Variables

Nate Rini<mailto:nate@schedmd.com> changed bug 15425<https://bugs.schedmd.com/show_bug.cgi?id=15425>
What
Removed
Added
Status
RESOLVED
OPEN
Resolution
INFOGIVEN
---
Comment # 6<https://bugs.schedmd.com/show_bug.cgi?id=15425#c6> on bug 15425<https://bugs.schedmd.com/show_bug.cgi?id=15425> from Nate Rini<mailto:nate@schedmd.com>

(In reply to Tom Ryan from comment #5<show_bug.cgi?id=15425#c5>)

> I have no further questions. My preference is to have an environment created

> when the slurm job is run through the API as opposed to sending all

> environment variables. This makes it difficult to send a job on behalf of a

> user.



Can you please clarify where this environment should be coming from? It is

generally expected slurmrestd will be started by systemd and won't have any

(useful) environment to pass along.

________________________________
You are receiving this mail because:

  *   You reported the bug.

The lack of XDG_environment variables also seem to prevent certain applications from starting up or from functioning correctly.

I’d like the slurmrest api to treat jobs submitted similar to the way sbatch or srun work in regards to creating all the variables a user might get in their environment when they login or submit a job.

From: bugs@schedmd.com <bugs@schedmd.com>
Sent: Monday, December 5, 2022 3:40 PM
To: Ryan, Thomas P <ryantp@corning.com>
Subject: [EXTERNAL]--[Bug 15425] Job Submit from REST API doesn't create User Environment Variables

Nate Rini<mailto:nate@schedmd.com> changed bug 15425<https://bugs.schedmd.com/show_bug.cgi?id=15425>
What
Removed
Added
Status
RESOLVED
OPEN
Resolution
INFOGIVEN
---
Comment # 6<https://bugs.schedmd.com/show_bug.cgi?id=15425#c6> on bug 15425<https://bugs.schedmd.com/show_bug.cgi?id=15425> from Nate Rini<mailto:nate@schedmd.com>

(In reply to Tom Ryan from comment #5<show_bug.cgi?id=15425#c5>)

> I have no further questions. My preference is to have an environment created

> when the slurm job is run through the API as opposed to sending all

> environment variables. This makes it difficult to send a job on behalf of a

> user.



Can you please clarify where this environment should be coming from? It is

generally expected slurmrestd will be started by systemd and won't have any

(useful) environment to pass along.

________________________________
You are receiving this mail because:

  *   You reported the bug.

(In reply to Tom Ryan from comment #7)
> I have a web application that is submitting a job on a user’s behalf to
> slurmrestd, the job will start TurboVNC for the user. TurboVNC cannot start
> because it’s missing environment variables such as $USER, $HOME, and more (I
> haven’t tracked down all of the variables yet)
> 
> Currently I’m submitting the jobs for the user using sudo -u <username>
> sbatch in order to carry over all the users environment variables in order
> to start the TurboVNC session. I’d like a cleaner way to submit the jobs.
> I’ve attempted to source files in order to create the environment for the
> users as part of the job submission script.

(In reply to Tom Ryan from comment #8)
> The lack of XDG_environment variables also seem to prevent certain
> applications from starting up or from functioning correctly.
> 
> I’d like the slurmrest api to treat jobs submitted similar to the way sbatch
> or srun work in regards to creating all the variables a user might get in
> their environment when they login or submit a job.

Looking into your request.

(In reply to Nate Rini from comment #9)
> (In reply to Tom Ryan from comment #7)
> > I have a web application that is submitting a job on a user’s behalf to
> > slurmrestd, the job will start TurboVNC for the user. TurboVNC cannot start
> > because it’s missing environment variables such as $USER, $HOME, and more (I
> > haven’t tracked down all of the variables yet)
> > 
> > Currently I’m submitting the jobs for the user using sudo -u <username>
> > sbatch in order to carry over all the users environment variables in order
> > to start the TurboVNC session. I’d like a cleaner way to submit the jobs.
> > I’ve attempted to source files in order to create the environment for the
> > users as part of the job submission script.
> 
> (In reply to Tom Ryan from comment #8)
> > The lack of XDG_environment variables also seem to prevent certain
> > applications from starting up or from functioning correctly.
> > 
> > I’d like the slurmrest api to treat jobs submitted similar to the way sbatch
> > or srun work in regards to creating all the variables a user might get in
> > their environment when they login or submit a job.
> 
> Looking into your request.

Please verify if the jobs have the "get_user_environment" field set to true at submission time.
> https://slurm.schedmd.com/rest_api.html#v0.0.38_job_properties

Any updates?

Sorry yes,

I was able to submit the job and start up a TurboVNC session. I had to define the environment variables for USER HOME PATH TEMP. Once the job was submitted I had to define more variables before TurboVNC would start up a GNOME Desktop session, see below. But all is working now

    export XDG_RUNTIME_DIR=$TEMP
    export XDG_SESSION_ID=$(</proc/self/sessionid)
    export XDG_SESSION_TYPE=tty
    export XDG_SESSION_CLASS=user

-Tom

From: bugs@schedmd.com <bugs@schedmd.com>
Sent: Wednesday, January 4, 2023 3:57 PM
To: Ryan, Thomas P <ryantp@corning.com>
Subject: [EXTERNAL]--[Bug 15425] Job Submit from REST API doesn't create User Environment Variables


Caution: This email originated from outside of Corning. Do not click links or open attachments unless you recognize the sender and know the content is safe. Report phishing by using the “PhishMe Reporter Button” above or forward the email to Phishyemails@corning.com<mailto:Phishyemails@corning.com>.
Comment # 11<https://bugs.schedmd.com/show_bug.cgi?id=15425#c11> on bug 15425<https://bugs.schedmd.com/show_bug.cgi?id=15425> from Nate Rini<mailto:nate@schedmd.com>

Any updates?

________________________________
You are receiving this mail because:

  *   You reported the bug.

(In reply to Tom Ryan from comment #12)
> I was able to submit the job and start up a TurboVNC session.

Was this job started with get_user_environment=true?

It was, however the “get_user_environment” doesn’t appear to add any additional variables. I’ve attempted to submit the job with and without “get_user_environment”, there isn’t a difference.

The jobs are submitted on the users behalf using a web application so there isn’t an environment to carry over. It sounds like it instead would need to “be created” similar to an ssh login, or login with gdm, when all those variables are set.

-Tom

From: bugs@schedmd.com <bugs@schedmd.com>
Sent: Monday, January 9, 2023 10:13 AM
To: Ryan, Thomas P <ryantp@corning.com>
Subject: [EXTERNAL]--[Bug 15425] Job Submit from REST API doesn't create User Environment Variables


Caution: This email originated from outside of Corning. Do not click links or open attachments unless you recognize the sender and know the content is safe. Report phishing by using the “PhishMe Reporter Button” above or forward the email to Phishyemails@corning.com<mailto:Phishyemails@corning.com>.
Comment # 13<https://bugs.schedmd.com/show_bug.cgi?id=15425#c13> on bug 15425<https://bugs.schedmd.com/show_bug.cgi?id=15425> from Nate Rini<mailto:nate@schedmd.com>

(In reply to Tom Ryan from comment #12<show_bug.cgi?id=15425#c12>)

> I was able to submit the job and start up a TurboVNC session.



Was this job started with get_user_environment=true?

________________________________
You are receiving this mail because:

  *   You reported the bug.

(In reply to Tom Ryan from comment #14)
> It was, however the “get_user_environment” doesn’t appear to add any
> additional variables. I’ve attempted to submit the job with and without
> “get_user_environment”, there isn’t a difference.
> 
> The jobs are submitted on the users behalf using a web application so there
> isn’t an environment to carry over. It sounds like it instead would need to
> “be created” similar to an ssh login, or login with gdm, when all those
> variables are set.

The variables listed in comment#12 are set via pam_systemd which is disabled for Slurm due it causing systemd to steal processes from Slurm's control. Full details are in bug#5920.

We have the workaround for the conflict documented here to get the similiar functionality:
> https://slurm.schedmd.com/pam_slurm_adopt.html#PAM_CONFIG

Applying the changes there along with using get_user_environment=true should fix the issue. Just setting the TaskProlog from the link above should resolve the environment variable issue.

(In reply to Nate Rini from comment #16)

Have these instructions helped?

(In reply to Nate Rini from comment #17)
> (In reply to Nate Rini from comment #16)
> 
> Have these instructions helped?

I'm going to time this ticket out. Please respond at your convenience and we can continue debugging.