Ticket 4770

We've noticed an odd issue when creating new accounts which we think is related to the length of the account name.

Having recently launched a new cluster we've switched to using account names with the format <PIsurname>-<servicelevel>-<type> where servicelevel is SL[1-4] and type is CPU, GPU, or KNL. At a minimum we'd expect to have <PIsurname>-SL3-CPU and <PIsurname>-SL4-CPU then optionally <PIsurname>-SL[3-4]-GPU and/or <PIsurname>-SL[34]-KNL and possibly paying SL1 or SL2 accounts for any, or all of, CPU, GPU, and KNL.

What we've found is that if we create a PI_TESTABCDE-FGHIJK (I've replaced actual PI's surname with TESTABCDE-FGHIJK but it was that long - a double-barrelled surname) account then a TESTABCDE-FGHIJK-SL3-CPU and TESTABCDE-FGHIJK-SL4-CPU account, each with PI_TESTABCDE-FGHIJK as their parent, sacctmgr then complains when we try and create a TESTABCDE-FGHIJK-SL3-GPU account. See below for various commands and output:

[root@slurm-master ~]# sacctmgr -vi add account Name=pi_testabcde-fghijk Description="Simon Flood" Cluster=csd3 parent=uis fairshare=parent
sacctmgr: Accounting storage SLURMDBD plugin loaded with AuthInfo=(null)
 Adding Account(s)
  pi_testabcde-fghijk
 Settings
  Description     = simon flood
  Organization    = Parent/Account Name
 Associations
  A = pi_testabc C = csd3
 Settings
  Fairshare     = parent
  Parent        = uis
[root@slurm-master ~]# sacctmgr -vi add account Name=TESTABCDE-FGHIJK-SL3-CPU GrpTRESMins=cpu=12000000 DefaultQOS=cpu2 QOS=cpu2,intr Cluster=csd3 parent=pi_testabcde-fghijk fairshare=0
sacctmgr: Accounting storage SLURMDBD plugin loaded with AuthInfo=(null)
 Adding Account(s)
  testabcde-fghijk-sl3-cpu
 Settings
  Description     = Account Name
  Organization    = Parent/Account Name
 Associations
  A = testabcde- C = csd3
 Settings
  Fairshare     = 0
  GrpTRESMins   = cpu=12000000
  Parent        = pi_testabcde-fghijk
  QOS           = cpu2,intr
  DefQOS        = cpu2
[root@slurm-master ~]# sacctmgr -vi add account Name=TESTABCDE-FGHIJK-SL4-CPU QOS=cpu3 Cluster=csd3 parent=pi_testabcde-fghijk fairshare=0
sacctmgr: Accounting storage SLURMDBD plugin loaded with AuthInfo=(null)
 Adding Account(s)
  testabcde-fghijk-sl4-cpu
 Settings
  Description     = Account Name
  Organization    = Parent/Account Name
 Associations
  A = testabcde- C = csd3
 Settings
  Fairshare     = 0
  Parent        = pi_testabcde-fghijk
  QOS           = cpu3
[root@slurm-master ~]# sacctmgr -vi add account Name=TESTABCDE-FGHIJK-SL3-GPU GrpTRESMins=gres/gpu=480000 DefaultQOS=gpu2 QOS=gpu2,intr Cluster=csd3 parent=pi_testabcde-fghijk fairshare=0
sacctmgr: Accounting storage SLURMDBD plugin loaded with AuthInfo=(null)
 Adding Account(s)
  testabcde-fghijk-sl3-gpu
 Settings
  Description     = Account Name
  Organization    = Parent/Account Name
 Associations
  A = testabcde- C = csd3
 Settings
  Fairshare     = 0
  GrpTRESMins   = gres/gpu=480000
  Parent        = pi_testabcde-fghijk
  QOS           = gpu2,intr
  DefQOS        = gpu2
 Problem adding accounts: Unspecified error
[root@slurm-master ~]# sacctmgr -n show account format=Account'%-25',Description'%-30',Organization'%-20' | grep -i testabcde-fghijk
pi_testabcde-fghijk       simon flood                    uis
testabcde-fghijk-sl3-cpu  testabcde-fghijk-sl3-cpu pi_testabcde-fghijk
testabcde-fghijk-sl4-cpu  testabcde-fghijk-sl4-cpu pi_testabcde-fghijk

When we originally saw this, trying to create the TESTABCDE-FGHIJK-SL3-GPU account gave an output suggesting it was trying to create an association rather than account but that didn't happen when repeating with fake "PI surname" for this message.

The other odd thing which we suspect is related is that when trying to undo these account additions (as we created them with shorter names) is that the delete deletes the association but not the actual accounts:

[root@slurm-master ~]# sacctmgr delete account name=testabcde-fghijk-sl3-cpu cluster=csd3
 Deleting account associations...
  C = csd3       A = testabcde-fghijk-sl3-cpu of pi_testabcde-fghijk
Would you like to commit changes? (You have 30 seconds to decide)
(N/y): y
[root@slurm-master ~]# sacctmgr delete account name=testabcde-fghijk-sl4-cpu cluster=csd3
 Deleting account associations...
  C = csd3       A = testabcde-fghijk-sl4-cpu of pi_testabcde-fghijk
Would you like to commit changes? (You have 30 seconds to decide)
(N/y): y
[root@slurm-master ~]# sacctmgr -n show account format=Account'%-25',Description'%-30',Organization'%-20' | grep -i testabcde-fghijk
pi_testabcde-fghijk       simon flood                    uis
testabcde-fghijk-sl3-cpu  testabcde-fghijk-sl3-cpu pi_testabcde-fghijk
testabcde-fghijk-sl4-cpu  testabcde-fghijk-sl4-cpu pi_testabcde-fghijk

If we then check the MySQL table it shows the accounts still exist but not associations. We're then tidying up by deleting the accounts manually in MySQL.

Our guess is that when creating the account sacctmgr is checking and comparing partial existing account names hence thinking there's a clash. I've had a quick look at the various bits of source code for sacctmgr but with my limited C knowledge haven't spotted anything obvious.

Previously we were using a mix of <PIsurname>-<servicelevel> for CPU and <PIsurname>-<servicelevel>-GPU for GPU (we didn't have KNL) so it's possible this issue existed in an earlier version of Slurm (we are using Slurm 14.11.8 on our old cluster) but we weren't hitting it.

Our new Slurm master is running Slurm 17.02.9 on Red Hat Enterprise Linux 7.3.

If you need further information please ask.

Regards,
Simon
-- 
Simon Flood
HPC System Administrator
University of Cambridge Information Services
United Kingdom