Created attachment 2979 [details]
Do not clear SLURM_UMASK env var

The SLURM_UMASK environment variable is being cleared rather than being visible to users. This one line patch removes the code to clear it (at least in version 15.08).

Since you think this is useful, I'll make this change plus matching documentation in version 16.05 (to be released May 2016).

Here is the commit in Slurm's version 16.05 code branch:
https://github.com/SchedMD/slurm/commit/58dea24602fe7b6cd98ae21df7780e13e6ae4d3a

(In reply to Moe Jette from comment #1)
> Created attachment 2979 [details]
> Do not clear SLURM_UMASK env var
> 
> The SLURM_UMASK environment variable is being cleared rather than being
> visible to users. This one line patch removes the code to clear it (at least
> in version 15.08).

There are a couple of downsides to this change (that we ran into now that we started testing 16.05):

  - When executed within an sbatch script, srun will always propagate the SLURM_UMASK setting no matter what kind of "ulimit" calls precede it. It is rather hard to debug such problems for users.
  - sbatch and salloc will behave differently since only the former one sets SLURM_UMASK.

Have these been taking into account?

re-opening the ticket, but I have no idea when I might have time to investigate

Created attachment 3588 [details]
Do not set SLURM_UMASK for batch jobs, srun command only

This patch (to Slurm version 16.05) will avoid setting SLURM_UMASK for batch jobs, but will continue to be set for anything spawned by an srun command.

Michael, will this still satisfy your requirements?

The specific case (that inspired the original bug) involves jobs submitted via slurm-drmaa (apps.man.poznan.pl/trac/slurm-drmaa).  So I guess I don't know how this change would affect us.

slurm-drmaa appears to create the job record using slurm primitives (is that the right term?)  slurm_init_job_desc_msg() seems to be one such "primitive" used for creating the job.  I don't know if that will bypass the check or if a job created vi drmaa is considered batch or not.  The slurm-drmaa code does attempt to propagate the local environment to the job, but I'm not sure that's a factor here either.

I certainly understand the issue that Dorian has highlighted- it is unclear in that circumstance which mask will be propagated to the srun job.  This is also likely the right fix as one can always set umask in the job script (or template in drmaa-speak).

There are other ways to fix our specific case, so I think I can say we'd be fine with it.  We'll just have to verify how these scripts work when we get to 16.

Thanks for checking in.

M

----- On Oct 13, 2016, at 8:00 AM,  bugs@schedmd.com wrote:

> https://bugs.schedmd.com/show_bug.cgi?id=2609
> 
> --- Comment #5 from Moe Jette <jette@schedmd.com> ---
> Created attachment 3588 [details]
>  --> https://bugs.schedmd.com/attachment.cgi?id=3588&action=edit
> Do not set SLURM_UMASK for batch jobs, srun command only
> 
> This patch (to Slurm version 16.05) will avoid setting SLURM_UMASK for batch
> jobs, but will continue to be set for anything spawned by an srun command.
> 
> Michael, will this still satisfy your requirements?
> 
> --
> You are receiving this mail because:
> You reported the bug.

OK, I'll mark this resolved unless I hear back from you.

(In reply to Michael Gutteridge from comment #6)
> The specific case (that inspired the original bug) involves jobs submitted
> via slurm-drmaa (apps.man.poznan.pl/trac/slurm-drmaa).  So I guess I don't
> know how this change would affect us.
> 
> slurm-drmaa appears to create the job record using slurm primitives (is that
> the right term?)  slurm_init_job_desc_msg() seems to be one such "primitive"
> used for creating the job.  I don't know if that will bypass the check or if
> a job created vi drmaa is considered batch or not.  The slurm-drmaa code
> does attempt to propagate the local environment to the job, but I'm not sure
> that's a factor here either.
> 
> I certainly understand the issue that Dorian has highlighted- it is unclear
> in that circumstance which mask will be propagated to the srun job.  This is
> also likely the right fix as one can always set umask in the job script (or
> template in drmaa-speak).

Slurm propagates the umask at submit time by setting the SLURM_UMASK environment variable. I would _guess_ that drmaa does so, in which case looking directly at the umask rather than the SLURM_UMASK environment variable would probably be best as that should be portable across various resource managers.