Ticket 5869

Summary: Access to a partition on a group or user basis
Product: Slurm Reporter: David Baker <d.j.baker>
Component: SchedulingAssignee: Director of Support <support>
Status: RESOLVED INFOGIVEN QA Contact:
Severity: 4 - Minor Issue    
Priority: ---    
Version: 18.08.0   
Hardware: Linux   
OS: Linux   
Site: OCF Slinky Site: ---
Alineos Sites: --- Atos/Eviden Sites: ---
Confidential Site: --- Coreweave sites: ---
Cray Sites: --- DS9 clusters: ---
Google sites: --- HPCnow Sites: ---
HPE Sites: --- IBM Sites: ---
NOAA SIte: --- NoveTech Sites: ---
Nvidia HWinf-CS Sites: --- OCF Sites: Southampton University
Recursion Pharma Sites: --- SFW Sites: ---
SNIC sites: --- Tzag Elita Sites: ---
Linux Distro: --- Machine Name:
CLE Version: Version Fixed:
Target Release: --- DevPrio: ---
Emory-Cloud Sites: ---

Description David Baker 2018-10-17 09:54:41 MDT
Hello,

In our slurm cluster we have defined a partition that can only be accessed by the members of one group called "lyceum". I'm starting a piece of work that potentially needs users to be members of two accounting group and as a test I thought I would add myself to the lyceum group -- I note that my primary group is "soton". I can see my group associations using the following command...

[root@blue51 slurm]# sacctmgr show associations | grep djb1
        i5     lyceum       djb1                    1                                                                                                                                                  normal                         
        i5      soton       djb1                    1                                                                                                                                             normal,test    normal

So, as djb1, I tried to submit a job to the lyceum partition defined as...

PartitionName=lyceum nodes=pink[51-60] Shared=Yes MaxCPUsPerNode=54 DefaultTime=02:00:00 MaxTime=60:00:00 QOS=lyceum State=UP AllowAccounts=lyceum

I used the submission command...

sinteractive -p lyceum --gres=gpu:1

Unfortunately, I didn't get permission to submit the job to the partition (despite being a member of the lyceum accounting group).The slurmctld log gave the following message....

[2018-10-17T16:41:00.128] _slurm_rpc_submit_batch_job: JobId=171840 InitPrio=54561 usec=3506
[2018-10-17T16:41:00.254] part_policy_valid_acct: Job's account not permitted to use this partition (lyceum allows lyceum not soton)

The user djb1 needs to be able to submit jobs to both the "public" partitions and the private lyceum partition. Could you please advise me? Can  I alternatively specify a list of users that access a partition instead of a group, for example?

Best regards,
David
Comment 1 Michael Hinton 2018-10-17 15:09:59 MDT
Hey David,

I think I know what your problem is. If no account is specified for a slurm job, the default account will be used. Your default account must be `soton`, because that was probably the first account you were put under.  So you need to explicitly specify the `lyceum` account in addition to specifying the `lyceum` partition.

`sinteractive` seems to be a custom wrapper command that we didn’t make. However, I did find this: https://www.massive.org.au/userguide/running-slurm-jobs/running-interactive-jobs. This page indicates that you can simply specify the account like so:

    sinteractive -p lyceum --account=lyceum --gres=gpu:1

Give this a try and let me know how it goes!

-Michael

P.S. Note that the `Shared` parameter has been discontinued in favor of `OverSubscribe`.
Comment 2 Michael Hinton 2018-10-17 15:29:46 MDT
To see the default account of users, simply do:

    sacctmgr show user

If you want, you can even change your default account to lyceum with the following command:

    sacctmgr modify user set defaultaccount=lyceum where name=djb1

See https://slurm.schedmd.com/accounting.html (though it appears that some of the commands are a little inaccurate - replace `default` with `defaultaccount`).
Comment 3 David Baker 2018-10-18 04:57:14 MDT
Hello,


Thank you. In other words I should make the lyceum group my default group to access the lyceum partition. I did that this morning and found that I could submit jobs to the lyceum partition and the public partitions which what we are looking for.


Please feel free to consider this ticket closed.


Best regards,

David

________________________________
From: bugs@schedmd.com <bugs@schedmd.com>
Sent: 17 October 2018 22:29
To: Baker D.J.
Subject: [Bug 5869] Access to a partition on a group or user basis


Comment # 2<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.schedmd.com%2Fshow_bug.cgi%3Fid%3D5869%23c2&data=01%7C01%7Cd.j.baker%40soton.ac.uk%7C23e3f32cf36e498dc96508d63477aabe%7C4a5378f929f44d3ebe89669d03ada9d8%7C1&sdata=UpQRaK8Ewttvn1QKFqFmtxuSnXZaQdhA8ftUt4FqGbo%3D&reserved=0> on bug 5869<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.schedmd.com%2Fshow_bug.cgi%3Fid%3D5869&data=01%7C01%7Cd.j.baker%40soton.ac.uk%7C23e3f32cf36e498dc96508d63477aabe%7C4a5378f929f44d3ebe89669d03ada9d8%7C1&sdata=Z6nre01rjcrn55gSlI%2BGhsKgrlAtMQa8mOHu2ojg80w%3D&reserved=0> from Michael Hinton<mailto:hinton@schedmd.com>

To see the default account of users, simply do:

    sacctmgr show user

If you want, you can even change your default account to lyceum with the
following command:

    sacctmgr modify user set defaultaccount=lyceum where name=djb1

See https://slurm.schedmd.com/accounting.html<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.schedmd.com%2Faccounting.html&data=01%7C01%7Cd.j.baker%40soton.ac.uk%7C23e3f32cf36e498dc96508d63477aabe%7C4a5378f929f44d3ebe89669d03ada9d8%7C1&sdata=%2BkqjS3QM0H6ycMnfnKwXgzABWRGPle89JeVbIWKEDcM%3D&reserved=0> (though it appears that some of
the commands are a little inaccurate - replace `default` with
`defaultaccount`).

________________________________
You are receiving this mail because:

  *   You reported the bug.
Comment 4 Michael Hinton 2018-10-18 10:00:42 MDT
(In reply to David Baker from comment #3)
> Thank you. In other words I should make the lyceum group my default group to
> access the lyceum partition.

To be clear, changing your user's default account to lyceum does NOT give you access to the partition. You were already given access when your user was added under the lyceum account. Rather, setting the default account to lyceum simply makes it so you don't have to specify `--account=lyceum` every time.

After changing defaultaccount=lyceum, the reason you didn't need to specify an account for the other public partitions is because they allow access from ALL accounts by default. Only the lyceum partition checks for the lyceum account (via `AllowAccounts=lyceum`). I hope that makes sense.

Closing out bug.

Cheers,
Michael