Ticket 16032

Summary: Requirement for user access to a reservation for an account
Product: Slurm Reporter: Renata Dart <renata>
Component: reservationsAssignee: Ben Roberts <ben>
Status: RESOLVED INFOGIVEN QA Contact:
Severity: 4 - Minor Issue    
Priority: ---    
Version: 22.05.2   
Hardware: Linux   
OS: Linux   
Site: SLAC Alineos Sites: ---
Atos/Eviden Sites: --- Confidential Site: ---
Coreweave sites: --- Cray Sites: ---
DS9 clusters: --- HPCnow Sites: ---
HPE Sites: --- IBM Sites: ---
NOAA SIte: --- OCF Sites: ---
Recursion Pharma Sites: --- SFW Sites: ---
SNIC sites: --- Linux Distro: ---
Machine Name: CLE Version:
Version Fixed: Target Release: ---
DevPrio: --- Emory-Cloud Sites: ---

Description Renata Dart 2023-02-14 19:26:41 MST
Hi SchedMD, we are not running with AccountingStorageEnforce and in ticket 15378
it was stated:

"There is no enforcement, so a user can specify any account even if
they do not have an association with that account."

And that does seem to be true.  If I create a user with no associated account
and then submit	a batch	job as that user specifying an account in the
batch submission script, the job runs and an scontrol of the job shows
that the account is the	one specified.

Is this supposed to also work for reservations?  That is, if you create
a reservation for a certain account such as:

scontrol create reservation account=lcls:cxilx8720 starttime="2023-02-14T15:42" Duration="20:00" tres=cpu=1 Partition=roma

Can a user with no associated account submit a job requesting that	reservation?	
I tried	it and it doesn't seem to work, permission denied.


Thanks,
Renata
Comment 1 Ben Roberts 2023-02-15 11:19:07 MST
Hi Renata,

It's true that without the AccountingStorageEnforce flag Slurm will allow users to specify any account they want and allow the jobs to run.  Reservations are slightly different though since it's reserving nodes for a specific type of job.  When you set an account as the access list for a reservation it will enforce that requirement for jobs to be able to run in the reservation.  

If you are looking for a way to allow more users to access a reservation you could specify an account that is denied access to the reservation, which would then implicitly allow all other accounts.  Here's an example of how that would look.

I have an account called 'sub6' and I say that jobs from that account are denied access to reservation.
$ scontrol create reservation accounts=-sub6 starttime=now duration=1:00:00 nodecnt=1
Reservation created: sub6_5

$ scontrol show reservations sub6_5 
ReservationName=sub6_5 StartTime=2023-02-15T11:57:09 EndTime=2023-02-15T12:57:09 Duration=01:00:00
   Nodes=node01 NodeCnt=1 CoreCnt=6 Features=(null) PartitionName=debug Flags=
   TRES=cpu=12
   Users=(null) Groups=(null) Accounts=-sub6 Licenses=(null) State=ACTIVE BurstBuffer=(null) Watts=n/a
   MaxStartDelay=(null)




If I submit a job to a reservation that is not defined on my system it will allow the job to run in that reservation.
$ sbatch -n1 -Afake --reservation=sub6_5 --wrap='srun sleep 30'
Submitted batch job 1454

$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              1454     debug     wrap    user4  R       0:00      1 node01




You should be aware if a user specifies the account that was excluded from the reservation, their job will be rejected.
$ sbatch -n1 -Asub6 --reservation=sub6_5 --wrap='srun sleep 30'
sbatch: error: Batch job submission failed: Access denied to requested reservation




I remember from the previous ticket that you were working on getting accounts created the way you wanted.  Is that still moving forward?  Let me know if you have any questions about the reservation example I showed and if that sounds like something that will work.

Thanks,
Ben
Comment 2 Renata Dart 2023-02-15 17:28:36 MST
Hi Ben, thanks for this clarification, it is what I was expecting
based on my tests.  The decision still hasn't been made to move
forward with AccountingStorageEnforce, so these kinds of questions are
cropping up.  

Renata



On Wed, 15 Feb 2023, bugs@schedmd.com wrote:

>https://bugs.schedmd.com/show_bug.cgi?id=16032
>
>--- Comment #1 from Ben Roberts <ben@schedmd.com> ---
>Hi Renata,
>
>It's true that without the AccountingStorageEnforce flag Slurm will allow users
>to specify any account they want and allow the jobs to run.  Reservations are
>slightly different though since it's reserving nodes for a specific type of
>job.  When you set an account as the access list for a reservation it will
>enforce that requirement for jobs to be able to run in the reservation.  
>
>If you are looking for a way to allow more users to access a reservation you
>could specify an account that is denied access to the reservation, which would
>then implicitly allow all other accounts.  Here's an example of how that would
>look.
>
>I have an account called 'sub6' and I say that jobs from that account are
>denied access to reservation.
>$ scontrol create reservation accounts=-sub6 starttime=now duration=1:00:00
>nodecnt=1
>Reservation created: sub6_5
>
>$ scontrol show reservations sub6_5 
>ReservationName=sub6_5 StartTime=2023-02-15T11:57:09
>EndTime=2023-02-15T12:57:09 Duration=01:00:00
>   Nodes=node01 NodeCnt=1 CoreCnt=6 Features=(null) PartitionName=debug Flags=
>   TRES=cpu=12
>   Users=(null) Groups=(null) Accounts=-sub6 Licenses=(null) State=ACTIVE
>BurstBuffer=(null) Watts=n/a
>   MaxStartDelay=(null)
>
>
>
>
>If I submit a job to a reservation that is not defined on my system it will
>allow the job to run in that reservation.
>$ sbatch -n1 -Afake --reservation=sub6_5 --wrap='srun sleep 30'
>Submitted batch job 1454
>
>$ squeue
>             JOBID PARTITION     NAME     USER ST       TIME  NODES
>NODELIST(REASON)
>              1454     debug     wrap    user4  R       0:00      1 node01
>
>
>
>
>You should be aware if a user specifies the account that was excluded from the
>reservation, their job will be rejected.
>$ sbatch -n1 -Asub6 --reservation=sub6_5 --wrap='srun sleep 30'
>sbatch: error: Batch job submission failed: Access denied to requested
>reservation
>
>
>
>
>I remember from the previous ticket that you were working on getting accounts
>created the way you wanted.  Is that still moving forward?  Let me know if you
>have any questions about the reservation example I showed and if that sounds
>like something that will work.
>
>Thanks,
>Ben
>
>-- 
>You are receiving this mail because:
>You reported the bug.
Comment 3 Ben Roberts 2023-02-16 08:46:48 MST
I understand.  Hopefully excluding an account will work better for what you're trying to do.  Let me know if you have any other questions about this or if we can close the ticket.

Thanks,
Ben
Comment 4 Renata Dart 2023-02-28 07:44:19 MST
Hi Ben, thanks for your help.  A somewhat related question....is it
possible for a slurm operator or admin to submit a job as someone
else, without sudo?

Renata

On Thu, 16 Feb 2023, bugs@schedmd.com wrote:

>https://bugs.schedmd.com/show_bug.cgi?id=16032
>
>--- Comment #3 from Ben Roberts <ben@schedmd.com> ---
>I understand.  Hopefully excluding an account will work better for what you're
>trying to do.  Let me know if you have any other questions about this or if we
>can close the ticket.
>
>Thanks,
>Ben
>
>-- 
>You are receiving this mail because:
>You reported the bug.
Comment 5 Ben Roberts 2023-03-01 11:38:18 MST
We do have a flag that is available for the root user to submit jobs as other users, but I'm afraid it's not an option for admins or operators.  
https://slurm.schedmd.com/sbatch.html#OPT_uid

When you attempt to use it as a non-root user it will print out this message:
$ sbatch --uid=user2 --wrap='srun sleep 30'
sbatch: error: --uid only permitted by root user


Thanks,
Ben
Comment 6 Ben Roberts 2023-04-07 09:43:45 MDT
Hi Renata,

I believe the information I sent addressed your questions and I haven't heard any follow up questions.  I'll go ahead and close this ticket.  Let us know if there's anything else we can do to help.

Thanks,
Ben