Ticket 6306

Summary: Enabling new partition for all users
Product: Slurm Reporter: Ahmed Essam ElMazaty <ahmed.mazaty>
Component: OtherAssignee: Felip Moll <fmoll>
Status: RESOLVED INFOGIVEN QA Contact:
Severity: 4 - Minor Issue    
Priority: --- CC: agilmoreno, asanchezg, fmoll
Version: 18.08.3   
Hardware: Linux   
OS: Linux   
See Also: https://bugs.schedmd.com/show_bug.cgi?id=3549
Site: KAUST Slinky Site: ---
Alineos Sites: --- Atos/Eviden Sites: ---
Confidential Site: --- Coreweave sites: ---
Cray Sites: --- DS9 clusters: ---
Google sites: --- HPCnow Sites: ---
HPE Sites: --- IBM Sites: ---
NOAA SIte: --- NoveTech Sites: ---
Nvidia HWinf-CS Sites: --- OCF Sites: ---
Recursion Pharma Sites: --- SFW Sites: ---
SNIC sites: --- Tzag Elita Sites: ---
Linux Distro: --- Machine Name:
CLE Version: Version Fixed:
Target Release: --- DevPrio: ---
Emory-Cloud Sites: ---
Attachments: slurm.conf
partitions.conf
slurmctld.log
slurmdbd.log
slurmdbd.log after raising log level
updated slurm.conf

Description Ahmed Essam ElMazaty 2019-01-01 06:09:48 MST
Hello,
I was trying to create a new partition containing some of our AMD blades. So I edited the configuration file and it seems the partition is ready to receive jobs 

 sinfo -p amd
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
amd          up 14-00:00:0     24  maint db211-02-[1-8],db211-12-[1-8],db211-22-[1-8]
amd          up 14-00:00:0      6 drain* db201-22-[2-7]


Then I started adding users to that partition with sacctmgr 
However when I tried to submit jobs on this partition the following error  appeared 
sbatch: error: Batch job submission failed: Invalid account or account/partition combination specified

I checked slurmctld logs and found the same message 
_slurm_rpc_submit_batch_job: Invalid account or account/partition combination specified

And I couldn't submit jobs on that partition until I restarted slurmctld. Is that the expected behaviour?? Should I restart slurm control daemon every time I try to add someone to that partition?

Best regards,
Ahmed
Comment 1 Felip Moll 2019-01-02 04:30:43 MST
(In reply to Ahmed Essam ElMazaty from comment #0)
> Hello,
> I was trying to create a new partition containing some of our AMD blades. So
> I edited the configuration file and it seems the partition is ready to
> receive jobs 
> 
>  sinfo -p amd
> PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
> amd          up 14-00:00:0     24  maint
> db211-02-[1-8],db211-12-[1-8],db211-22-[1-8]
> amd          up 14-00:00:0      6 drain* db201-22-[2-7]
> 
> 
> Then I started adding users to that partition with sacctmgr 
> However when I tried to submit jobs on this partition the following error 
> appeared 
> sbatch: error: Batch job submission failed: Invalid account or
> account/partition combination specified
> 
> I checked slurmctld logs and found the same message 
> _slurm_rpc_submit_batch_job: Invalid account or account/partition
> combination specified
> 
> And I couldn't submit jobs on that partition until I restarted slurmctld. Is
> that the expected behaviour?? Should I restart slurm control daemon every
> time I try to add someone to that partition?
> 
> Best regards,
> Ahmed

Hi Ahmed,

Modifying allowed users/accounts of partitions in slurm.conf or adding a new
partition just needs an scontrol reconfig.

Can you show me how you have defined the partition in slurm.conf?

Just some notes:

You say you're "adding users to a partition using sacctmgr", what you're really doing
here is to create an association between a cluster/account/user/partition, that means
that this association will be allowed to run on the specified partition. For example
association with user "foo", account "bar", partition "xyz" means that user foo under
account bar (may have multiple accounts) will be able only to run under xyz partition.

This doesn't really "allow" a user to run on a partition if in the partition definition
the user is not allowed.

###### EXAMPLE1
i.e. a definition in slurm.conf of:

PartitionName=amd Nodes=db201-22-[2-7] Default=No State=Up

and an association of:

<cluster:x, account:y, user:z, partition:''> will be able to run on any partition including 'amd'.
<cluster:x, account:y, user:z, partition:'non-amd'> will NOT be able to run on 'amd'.
<cluster:x, account:y, user:z, partition:'amd'> will be able to run ONLY on 'amd'.

####### EXAMPLE2

i.e. a definition in slurm.conf of:

PartitionName=amd Nodes=db201-22-[2-7] Default=No State=Up AllowedAccounts=ZZZ

and an association of:

<cluster:x, account:y, user:z, partition:'[amd|non-amd|""]'>  Will NEVER be able to run on 'amd'.
<cluster:x, account:ZZZ, user:z, partition:'[amd|non-amd|""]'>  Will follow the rules of the previous example.
Comment 3 Ahmed Essam ElMazaty 2019-01-02 04:48:39 MST
Hello Felip


> 
> Modifying allowed users/accounts of partitions in slurm.conf or adding a new
> partition just needs an scontrol reconfig.
> 
scontrol reconfig didn't work for me.. I had to restart slurmctld with systemctl to avoid ' Invalid account/partition' error  

> Can you show me how you have defined the partition in slurm.conf?

That how I defined the partition
PartitionName=amd State=Up DefMemPerCPU=2048 Default=No MaxCPUsPerNode=64 Nodes=db213-[02,12,22]-[1,3-8],db213-[12,22]-[2],db214-[02,12]-[1-8] PriorityTier=100 MaxTime=14-00:00:00

> 
> Just some notes:
> 
> You say you're "adding users to a partition using sacctmgr", what you're
> really doing

I'm trying to add users using this command 
 sacctmgr -i add user Name=${user} DefaultAccount=default AdminLevel=None  Cluster=${CLUSTER} Account=default partitions=amd
Comment 5 Felip Moll 2019-01-02 05:12:49 MST
> I'm trying to add users using this command 
>  sacctmgr -i add user Name=${user} DefaultAccount=default AdminLevel=None 
> Cluster=${CLUSTER} Account=default partitions=amd

If you don't specify "partitions=amd" here, does it work?

Are the nodes new nodes or were it already in the system?
Comment 6 Felip Moll 2019-01-02 05:15:40 MST
(In reply to Felip Moll from comment #5)
> > I'm trying to add users using this command 
> >  sacctmgr -i add user Name=${user} DefaultAccount=default AdminLevel=None 
> > Cluster=${CLUSTER} Account=default partitions=amd
> 
> If you don't specify "partitions=amd" here, does it work?
> 
> Are the nodes new nodes or were it already in the system?

Ah, also note that when submitting a job the user will need to specify
the partition by hand, since the partition is not the Default one.

so this:

srun hostname <-- will fail
srun --partition=amd hostname <-- should work
Comment 7 Ahmed Essam ElMazaty 2019-01-02 05:19:31 MST

> If you don't specify "partitions=amd" here, does it work?

Actually users are already there and the can submit on other partitions
So it does not change anything if I removed it
> 
> Are the nodes new nodes or were it already in the system?
No those nodes were in the default patition  and we decided to move them to a new one
Comment 8 Ahmed Essam ElMazaty 2019-01-02 05:21:24 MST
> 
> Ah, also note that when submitting a job the user will need to specify
> the partition by hand, since the partition is not the Default one.
> 
> so this:
> 
> srun hostname <-- will fail
> srun --partition=amd hostname <-- should work

Actually if I didn't specify  '--partition' my job would run on the default partition. I can do that without problems.
But I could not submit jobs on the new one
Comment 9 Felip Moll 2019-01-02 05:37:35 MST
(In reply to Ahmed Essam ElMazaty from comment #7)
> 
> > If you don't specify "partitions=amd" here, does it work?
> 
> Actually users are already there and the can submit on other partitions
> So it does not change anything if I removed it

Well, this test will determine if adding an association without restrictions
will be able to run in the amd partition.

sacctmgr -i add user Name=${user} DefaultAccount=default AdminLevel=None Cluster=${CLUSTER} Account=default

and then with the new unrestricted assoc:

srun --partition=amd hostname

I don't know if your users/assocs are all restricted to specific partitions.
This was the reason I asked.

Secondly, I guess it is not the case, but just to confirm: are your UIDs changed?
Are you using files or other system (nss, sssd, ldap) for user resolution?

One other possible issue. Slurmctld mantains a cache of the info stored in the database.
If you add new info to the database with sacctmgr and for some reason slurmdbd cannot
contact with slurmctld this could case similar errors. Do you have any firewall between
slurmdbd/ctld? Run scontrol show assoc_mgr to confirm data is well seen by ctld before
restarting. I can't remember if you have dbd it in the same host than ctld or not.


If none of these ideas explain what you're seeing, please provide me with slurm.conf, slurmdbd and
slurmctld logs.

Thanks
Comment 10 Ahmed Essam ElMazaty 2019-01-02 05:48:17 MST
> Well, this test will determine if adding an association without restrictions
> will be able to run in the amd partition.
> 
> sacctmgr -i add user Name=${user} DefaultAccount=default AdminLevel=None
> Cluster=${CLUSTER} Account=default
> 
> and then with the new unrestricted assoc:
> 
> srun --partition=amd hostname
> 
> I don't know if your users/assocs are all restricted to specific partitions.
> This was the reason I asked.

That didn't work .. same error even after running scontrol reconfig


> 
> Secondly, I guess it is not the case, but just to confirm: are your UIDs
> changed?
> Are you using files or other system (nss, sssd, ldap) for user resolution?
> 

No IDs did not change 
and users can work normally on the other old partitions but not this one

> One other possible issue. Slurmctld mantains a cache of the info stored in
> the database.
> If you add new info to the database with sacctmgr and for some reason
> slurmdbd cannot
> contact with slurmctld this could case similar errors. Do you have any
> firewall between
> slurmdbd/ctld? Run scontrol show assoc_mgr to confirm data is well seen by
> ctld before
> restarting. I can't remember if you have dbd it in the same host than ctld
> or not.
> 

Slurmdbd and Slurmctld are on the same host 
scontrol show assoc_mgr was executed without problems
Comment 11 Ahmed Essam ElMazaty 2019-01-02 05:59:21 MST
Created attachment 8803 [details]
slurm.conf

Hello Felip
I will attach the files you requested
Here is slurm.conf
Comment 12 Ahmed Essam ElMazaty 2019-01-02 06:00:15 MST
Created attachment 8804 [details]
partitions.conf

partitions.conf
Comment 13 Ahmed Essam ElMazaty 2019-01-02 06:00:55 MST
Created attachment 8805 [details]
slurmctld.log
Comment 14 Ahmed Essam ElMazaty 2019-01-02 06:01:36 MST
Created attachment 8806 [details]
slurmdbd.log
Comment 15 Felip Moll 2019-01-02 06:06:18 MST
Thanks for the files.

> That didn't work .. same error even after running scontrol reconfig
> 

That's not expected.

> Slurmdbd and Slurmctld are on the same host 
> scontrol show assoc_mgr was executed without problems

Ok, have you seen the new association in this output?

Can you send me back the 'scontrol show assoc_mgr' output too at the time
an user is not able to submit? Tell me also what's the user that cannot run.
Comment 16 Ahmed Essam ElMazaty 2019-01-02 06:12:31 MST
> 
> Ok, have you seen the new association in this output?
> 
> Can you send me back the 'scontrol show assoc_mgr' output too at the time
> an user is not able to submit? Tell me also what's the user that cannot run.

The association is there for users that were added before restarting slurmctld 
[root@slurm209-02 ~]# scontrol show assoc_mgr| grep mazatyae
UserName=mazatyae(167627) DefAccount=default DefWckey= AdminLevel=None
ClusterName=dragon Account=default UserName=mazatyae(167627) Partition=amd ID=5012
ClusterName=dragon Account=default UserName=mazatyae(167627) Partition=batch ID=4402
ClusterName=dragon Account=default UserName=mazatyae(167627) Partition=debug ID=4403


However it's not there for users whom were added to the partition after restarting slurmctld
UserName=hafezpa(165422) DefAccount=default DefWckey= AdminLevel=None
ClusterName=dragon Account=default UserName=hafezpa(165422) Partition=batch ID=4248
ClusterName=dragon Account=default UserName=hafezpa(165422) Partition=debug ID=4249
Comment 17 Felip Moll 2019-01-02 06:30:32 MST
(In reply to Ahmed Essam ElMazaty from comment #16)
> The association is there for users that were added before restarting
> slurmctld 
> [root@slurm209-02 ~]# scontrol show assoc_mgr| grep mazatyae
> UserName=mazatyae(167627) DefAccount=default DefWckey= AdminLevel=None
> ClusterName=dragon Account=default UserName=mazatyae(167627) Partition=amd
> ID=5012
> ClusterName=dragon Account=default UserName=mazatyae(167627) Partition=batch
> ID=4402
> ClusterName=dragon Account=default UserName=mazatyae(167627) Partition=debug
> ID=4403
> 
> 
> However it's not there for users whom were added to the partition after
> restarting slurmctld
> UserName=hafezpa(165422) DefAccount=default DefWckey= AdminLevel=None
> ClusterName=dragon Account=default UserName=hafezpa(165422) Partition=batch
> ID=4248
> ClusterName=dragon Account=default UserName=hafezpa(165422) Partition=debug
> ID=4249

So the current situation is like that:

sacctmgr show assoc -pn|grep -i hafezpa : It shows one association with the partition=amd
scontrol show assoc_mgr |grep -i hafezpa : It DOESN'T show any association with the partition=amd

and a slurmctld restart seems to fix it, right?

If this is the case dbd and ctld are out-of-sync. Unfortunately the logs from
slurmdbd are not showing me anything. May you increase slurmdbd log level to debug2,
restart and then add a new association? Let's see if we find any problem while sending
back the info to ctld.
Comment 18 Ahmed Essam ElMazaty 2019-01-02 23:09:04 MST
> 
> So the current situation is like that:
> 
> sacctmgr show assoc -pn|grep -i hafezpa : It shows one association with the
> partition=amd
> scontrol show assoc_mgr |grep -i hafezpa : It DOESN'T show any association
> with the partition=amd
> 
> and a slurmctld restart seems to fix it, right?
> 
> If this is the case dbd and ctld are out-of-sync. Unfortunately the logs from
> slurmdbd are not showing me anything. May you increase slurmdbd log level to
> debug2,
> restart and then add a new association? Let's see if we find any problem
> while sending
> back the info to ctld.


I added a new user (ghattara)
executed scontrol reconfig
And nothing changed 

[root@slurm209-02 ~]# scontrol show assoc_mgr |grep -i ghattara
UserName=ghattara(170337) DefAccount=default DefWckey= AdminLevel=None
ClusterName=dragon Account=default UserName=ghattara(170337) Partition=batch ID=4978
ClusterName=dragon Account=default UserName=ghattara(170337) Partition=debug ID=4979

[root@slurm209-02 ~]# sacctmgr show assoc -pn|grep -i ghattara
dragon|default|ghattara|amd|1||||||||||||normal|||
dragon|default|ghattara|batch|1||||||||||||normal|||
dragon|default|ghattara|debug|1||||||||||||normal|||

I will attach slurmdbd logs after raising log level to debug2
Comment 19 Ahmed Essam ElMazaty 2019-01-02 23:09:49 MST
Created attachment 8820 [details]
slurmdbd.log after raising log level
Comment 21 Felip Moll 2019-01-03 02:47:29 MST
(In reply to Ahmed Essam ElMazaty from comment #19)
> Created attachment 8820 [details]
> slurmdbd.log after raising log level

That's really odd because the transaction seems to work correctly
and slurmdbd is sending the notification to slurmctld:

[2019-01-03T09:03:24.539] debug:  sending updates to dragon at 10.109.151.7(6917) ver 8448

I will come back to you when I have more information or tests to do.

Thanks
Comment 22 Felip Moll 2019-01-03 03:30:22 MST
This won't be related since part of the association was propagated,
but anyway can you tell me what's defined in slurmdbd.conf for the DbdHost and Addr?
Is it slurm209-02 ? Does it resolve to the same ip address than dm308-17?

slurm.conf:
ControlMachine=dm308-17
ControlAddr=dm308-17.ibex.kaust.edu.sa
ClusterName=dragon
SlurmctldPort=6917-6964
AccountingStorageType=accounting_storage/slurmdbd
AccountingStorageHost=dm308-17

slurmdbd log:
[2019-01-03T08:58:15.582] debug2: CommitDelay       = 1
[2019-01-03T08:58:15.583] debug2: DbdAddr           = slurm209-02
[2019-01-03T08:58:15.584] debug2: DbdBackupHost     = (null)
[2019-01-03T08:58:15.586] debug2: DbdHost           = slurm209-02
[2019-01-03T08:58:15.587] debug2: DbdPort           = 6819

[2019-01-03T09:03:24.539] debug:  sending updates to dragon at 10.109.151.7(6917) ver 8448
Comment 23 Ahmed Essam ElMazaty 2019-01-03 04:45:10 MST
Created attachment 8822 [details]
updated slurm.conf


Hello Felip, 
Sorry for that, I uploaded an old version from slurm.conf. Attached is the current one
Everything now is on slurm209-02
Comment 24 Felip Moll 2019-01-03 05:36:20 MST
(In reply to Ahmed Essam ElMazaty from comment #23)
> Created attachment 8822 [details]
> updated slurm.conf
> 
> 
> Hello Felip, 
> Sorry for that, I uploaded an old version from slurm.conf. Attached is the
> current one
> Everything now is on slurm209-02

No problem.

May you send also the chunk of ctld logs from [2019-01-03T08:58:09.593] to [2019-01-03T09:07:35.898] ?
(or all of it, no problem).
Comment 25 Ahmed Essam ElMazaty 2019-01-03 05:57:42 MST
> 
> No problem.
> 
> May you send also the chunk of ctld logs from [2019-01-03T08:58:09.593] to
> [2019-01-03T09:07:35.898] ?
> (or all of it, no problem).

Hello Felip,
Here are slurmctld logs during that period 

[2019-01-03T08:58:11.831] error: slurm_persist_conn_open_without_init: failed to open persistent connection to slurm209-02:6819: Connection refused
[2019-01-03T08:58:11.833] error: slurmdbd: Failure sending message: 11: Connection refused
[2019-01-03T08:58:21.377] error: slurmdbd: Sending PersistInit msg: Connection refused
[2019-01-03T08:58:31.421] error: slurmdbd: Sending PersistInit msg: Connection refused
[2019-01-03T08:58:41.996] error: slurmdbd: Sending PersistInit msg: Connection refused
[2019-01-03T08:58:48.623] error: slurmdbd: Sending PersistInit msg: Connection refused
[2019-01-03T08:58:51.633] error: slurmdbd: Sending PersistInit msg: Connection refused
[2019-01-03T08:59:01.268] error: slurmdbd: Sending PersistInit msg: Connection refused
[2019-01-03T08:59:11.867] error: slurmdbd: Sending PersistInit msg: Connection refused
[2019-01-03T08:59:22.074] error: slurmdbd: Sending PersistInit msg: Connection refused
[2019-01-03T08:59:32.745] error: slurmdbd: Sending PersistInit msg: Connection refused
[2019-01-03T08:59:42.771] error: slurmdbd: Sending PersistInit msg: Connection refused
[2019-01-03T08:59:53.270] error: slurmdbd: Sending PersistInit msg: Connection refused
[2019-01-03T09:00:03.375] error: slurmdbd: Sending PersistInit msg: Connection refused
[2019-01-03T09:00:13.669] error: slurmdbd: Sending PersistInit msg: Connection refused
[2019-01-03T09:00:23.228] error: slurmdbd: Sending PersistInit msg: Connection refused
[2019-01-03T09:00:33.741] error: slurmdbd: Sending PersistInit msg: Connection refused
[2019-01-03T09:00:34.551] error: Registration request from non-super user uid=600
[2019-01-03T09:00:52.485] update_node: node cn603-11-l state set to ALLOCATED
[2019-01-03T09:02:49.724] update_node: node cn603-11-l reason set to: NHC: check_ps_loadavg:  1-minute load average too high:  671 >= 320
[2019-01-03T09:02:49.725] update_node: node cn603-11-l state set to DRAINING
[2019-01-03T09:03:24.547] error: Update Association request from non-super user uid=600
[2019-01-03T09:03:40.108] job_submit.lua: Not touching jobs with reservation 'pathogen'
[2019-01-03T09:03:40.109] job_submit.lua: slurm_job_submit: features = nogpu
[2019-01-03T09:03:40.117] sched: _slurm_rpc_allocate_resources JobId=860305 NodeList=(null) usec=14468
[2019-01-03T09:03:40.688] sched: Allocate JobId=860305 NodeList=besest514-03 #CPUs=24 Partition=batch
[2019-01-03T09:03:42.320] _job_complete: JobId=794943 WEXITSTATUS 0
[2019-01-03T09:03:42.322] _job_complete: JobId=794943 done
[2019-01-03T09:03:46.789] Processing RPC: REQUEST_RECONFIGURE from uid=0
[2019-01-03T09:03:46.948] layouts: no layout to initialize
[2019-01-03T09:03:46.975] partition debug DefaultTime exceeds MaxTime (5760 > 120)
[2019-01-03T09:03:47.015] error: WARNING: switches lack access to 20 nodes: gpu504-37,gpu601-07,kccn104-02-[02-10,12-20]
[2019-01-03T09:03:47.015] TOPOLOGY: warning -- no switch can reach all nodes through its descendants.Do not use route/topology
[2019-01-03T09:03:47.081] error: Unable to resolve "db510-11-1": Unknown host
[2019-01-03T09:03:47.083] error: _set_slurmd_addr: failure on db510-11-1
[2019-01-03T09:03:47.089] error: Unable to resolve "db510-11-2": Unknown host
[2019-01-03T09:03:47.090] error: _set_slurmd_addr: failure on db510-11-2
[2019-01-03T09:03:47.166] restoring original state of nodes
[2019-01-03T09:03:47.176] cons_res: select_p_node_init
[2019-01-03T09:03:47.178] cons_res: preparing for 10 partitions
[2019-01-03T09:03:47.348] No parameter for mcs plugin, default values set
[2019-01-03T09:03:47.350] mcs: MCSParameters = (null). ondemand set.
[2019-01-03T09:03:54.826] SchedulerParameters=default_queue_depth=1024,max_rpc_cnt=0,max_sched_time=20,partition_job_depth=0,sched_max_job_start=0,sched_min_interval=2
[2019-01-03T09:03:57.524] _slurm_rpc_reconfigure_controller: completed usec=10894778
[2019-01-03T09:05:13.378] job_submit.lua: slurm_job_submit: features = nolmem
[2019-01-03T09:05:13.379] job_submit.lua: Memory per CPU is specified - not processing
[2019-01-03T09:05:13.380] job_submit.lua: slurm_job_submit: features = nolmem&nogpu
[2019-01-03T09:05:13.382] _job_create: invalid account or partition for user 170337, account 'default', and partition 'amd'
[2019-01-03T09:05:13.383] _slurm_rpc_allocate_resources: Invalid account or account/partition combination specified
[2019-01-03T09:05:20.619] job_submit.lua: slurm_job_submit: features = nolmem
[2019-01-03T09:05:20.621] job_submit.lua: Memory per CPU is specified - not processing
[2019-01-03T09:05:20.623] job_submit.lua: slurm_job_submit: features = nolmem&nogpu
[2019-01-03T09:05:20.624] _job_create: invalid account or partition for user 170337, account 'default', and partition 'amd'
[2019-01-03T09:05:20.625] _slurm_rpc_allocate_resources: Invalid account or account/partition combination specified
[2019-01-03T09:05:52.054] update_node: node cn603-11-l reason set to: NHC: check_ps_loadavg:  1-minute load average too high:  758 >= 320
[2019-01-03T09:05:52.055] update_node: node cn603-11-l state set to DRAINING
[2019-01-03T09:06:51.308] update_node: node cn603-11-l reason set to: NHC: check_ps_loadavg:  1-minute load average too high:  760 >= 320
[2019-01-03T09:06:51.309] update_node: node cn603-11-l state set to DRAINING
[2019-01-03T09:07:28.240] job_submit.lua: slurm_job_submit: features = nolmem
[2019-01-03T09:07:28.241] job_submit.lua: Memory per CPU is specified - not processing
Comment 26 Felip Moll 2019-01-03 06:15:46 MST
Running slurmctld as root and slurmdbd as slurm user (uid=600) seems to be causing the issue.

> [2019-01-03T09:03:24.547] error: Update Association request from non-super
> user uid=600

The message comes from slurmctld proc_req.c:_slurm_rpc_accounting_update_msg()->validate_super_user().

This function accepts the update association request for:
root (uid 0)
SlurmUser (uid 0 in your ctld)
an Administrator

So if you want to keep running ctld as root, you should set the admin level of 'slurm' to AdminLevel=administrator.

Please, try this:
sacctmgr update user slurm set adminlevel=administrator

You can check current level with
sacctmgr show user -p


After giving permissions to slurm user, please, retry to create a new user/association.
Comment 27 Ahmed Essam ElMazaty 2019-01-03 06:27:05 MST
(In reply to Felip Moll from comment #26)
> Running slurmctld as root and slurmdbd as slurm user (uid=600) seems to be
> causing the issue.
> 
> > [2019-01-03T09:03:24.547] error: Update Association request from non-super
> > user uid=600
> 
> The message comes from slurmctld
> proc_req.c:_slurm_rpc_accounting_update_msg()->validate_super_user().
> 
> This function accepts the update association request for:
> root (uid 0)
> SlurmUser (uid 0 in your ctld)
> an Administrator
> 
> So if you want to keep running ctld as root, you should set the admin level
> of 'slurm' to AdminLevel=administrator.
> 
> Please, try this:
> sacctmgr update user slurm set adminlevel=administrator
> 
> You can check current level with
> sacctmgr show user -p
> 
> 
> After giving permissions to slurm user, please, retry to create a new
> user/association.


Actually slurm is a local user on slurm node 
so sacctmgr show user -p does not show anything 
and sacctmgr update user slurm set adminlevel=administrator does not update anything
Comment 28 Felip Moll 2019-01-03 06:33:25 MST
> Actually slurm is a local user on slurm node 
> so sacctmgr show user -p does not show anything 
> and sacctmgr update user slurm set adminlevel=administrator does not update
> anything

Then you have to add the user to Slurm.

sacctmgr add -i account slurm 
sacctmgr add -i user slurm adminlevel=admin account=slurm


Or you can change the SlurmUser=root to slurm, or change slurmdbd to run as root.

But I thing the best way is creating this user in the database.

Why are you running slurmctld as root?
Comment 29 Ahmed Essam ElMazaty 2019-01-03 06:56:48 MST
(In reply to Felip Moll from comment #28)
> > Actually slurm is a local user on slurm node 
> > so sacctmgr show user -p does not show anything 
> > and sacctmgr update user slurm set adminlevel=administrator does not update
> > anything
> 
> Then you have to add the user to Slurm.
> 
> sacctmgr add -i account slurm 
> sacctmgr add -i user slurm adminlevel=admin account=slurm
> 
> 
> Or you can change the SlurmUser=root to slurm, or change slurmdbd to run as
> root.
> 
> But I thing the best way is creating this user in the database.
> 
> Why are you running slurmctld as root?


I created the user, set it as admin and executed scontrol reconfig
That didn't work 
Associations still do not appear correctly
I checked slurmctld.log and that error is still there
error: Update Association request from non-super user uid=600
Comment 30 Felip Moll 2019-01-03 07:05:19 MST
(In reply to Ahmed Essam ElMazaty from comment #29)
> (In reply to Felip Moll from comment #28)
> > > Actually slurm is a local user on slurm node 
> > > so sacctmgr show user -p does not show anything 
> > > and sacctmgr update user slurm set adminlevel=administrator does not update
> > > anything
> > 
> > Then you have to add the user to Slurm.
> > 
> > sacctmgr add -i account slurm 
> > sacctmgr add -i user slurm adminlevel=admin account=slurm
> > 
> > 
> > Or you can change the SlurmUser=root to slurm, or change slurmdbd to run as
> > root.
> > 
> > But I thing the best way is creating this user in the database.
> > 
> > Why are you running slurmctld as root?
> 
> 
> I created the user, set it as admin and executed scontrol reconfig
> That didn't work 
> Associations still do not appear correctly
> I checked slurmctld.log and that error is still there
> error: Update Association request from non-super user uid=600

Hm, that must be because adding the new slurm user also needs slurmdbd
to send the info to slurmctld, so there's a catch-22.

Restarting slurmctld now should make the slurm user appear in:

scontrol show assoc_mgr

and enable further modifications, since when starting slurmctld it will contact
slurmdbd the other way.

Can you try to restart slurmctld and show me the 'scontrol show assoc_mgr' for slurm
user?

Then try to create again a new assoc and see if the error persists.
Comment 31 Ahmed Essam ElMazaty 2019-01-03 12:45:50 MST
> 
> Hm, that must be because adding the new slurm user also needs slurmdbd
> to send the info to slurmctld, so there's a catch-22.
> 
> Restarting slurmctld now should make the slurm user appear in:
> 
> scontrol show assoc_mgr
> 
> and enable further modifications, since when starting slurmctld it will
> contact
> slurmdbd the other way.
> 
> Can you try to restart slurmctld and show me the 'scontrol show assoc_mgr'
> for slurm
> user?
> 
> Then try to create again a new assoc and see if the error persists.


After restarting slurmctld it finally worked!!
Thanks Felip for your help and efforts
So the problem was that slurm user did not have permission to write to slurmdbd??
Comment 32 Felip Moll 2019-01-03 14:47:03 MST
> After restarting slurmctld it finally worked!!
> Thanks Felip for your help and efforts
> So the problem was that slurm user did not have permission to write to
> slurmdbd??

Yes, that was it: since you are running the daemons with different users,
all of which are not SlurmUser (of the slurmctld), or root, must be added
to the accounting.

I will see if I can put a note on the documentation before closing the bug.

Glad it helped.

Regards,
Felip
Comment 33 Felip Moll 2019-01-04 06:59:36 MST
Small note has been added to documentation in commit 95495e4, available
in next 19.05.

Heading to close the issue now.