Ticket 8360

Created attachment 12786 [details]
slurm.conf

Hello,

We attempted to go to version 18.08.9 but ran into the following issue with the slurmdbd:

```
[2020-01-21T10:15:53.253] error: mysql_query failed: 1280 Incorrect index name 'type'
alter table tres_table modify `creation_time` bigint unsigned not null, modify `deleted` tinyint default 0 not null, modify `id` int not null auto_increment, modify `type` tinytext not null, modify `name` tinytext not null default '', drop primary key, add primary key (id), drop index type, add unique index (type(20), name(20));
[2020-01-21T10:15:53.253] error: Couldn't load specified plugin name for accounting_storage/mysql: Plugin init() callback failed
[2020-01-21T10:15:53.253] error: cannot create accounting_storage context for accounting_storage/mysql
[2020-01-21T10:15:53.253] fatal: Unable to initialize accounting_storage/mysql accounting storage plugin
```

We have since reverted to 18.08.8

David

Could you please get the backtrace from the coredump?

In gdb, run:

bt
bt full
thread apply all bt
thread apply all bt full


and upload the results as an attachment

Can you also tell us what change you made to your slurm.conf? Either a patch or prose description will do.

Created attachment 12788 [details]
gdb outputs

Gavin,

The change we made today was adding AllocNodes to the standard-oc partition:

```
PartitionName=standard-oc AllowAccounts=ALL QoS=standard-oc AllocNodes=gl-campus-login,gl-build,gl[3002-3010] Nodes=gl[3002-3010] TRESBillingWeights=cpu=43.03240741,mem=6.147486772G
```

Users needed to be able to submit to this partition from the compute nodes therein as part of their work flow.

David

Created attachment 12789 [details]
bandaid - should make slurmctld start

This looks like something we've seen before (bug 8285). Here's the patch from that bug.

Can you please apply this and restart slurmctld? Let us know if it starts up again or not.

This bug and 8285 look like a duplicate of 6739. However, you're already on 18.08.8 and the fix for 6739 (commit 4c48a84a6edb) is in 18.08.8, so it makes me wonder if there's another very similar bug.

Anyway, this patch should get the slurmctld running.

Marshall,

Thanks for the quick turn around.

We're looking at the source files for 18.08.8 and 18.08.9 and don't see the patched code in `src/common/bitstring.c` (lines 1397-1399). In other words, I'm not seeing the new lines the patch would add. So, not sure if this patch is just a stop gap or if SchedMD meant it to go into the source as an official thing, but wanted to point that out. It appears that it's not in 19.05 either, according to my colleague.

Just wanted to note that.

David

Marshall et al,

When attempting to apply the patch provided to 18.08.8 source and compiling we're seeing:

```
bitstring.c: In function 'bit_unfmt_hexmask':
bitstring.c:1398:10: error: 'SLURM_SUCCESS' undeclared (first use in this function)
   return SLURM_SUCCESS;
          ^
bitstring.c:1398:10: note: each undeclared identifier is reported only once for each function it appears in
```

David

Sorry, my mistake.

Replace SLURM_SUCCESS with 0 in the patch, and it will compile

Marshall et al,

The patch worked. After being applied, the patch enabled us to see the following:

```
[2020-01-21T11:44:41.089] fatal: Invalid node names in partition debug

```

in the slurmctld.log.

Once we fixed the issue with the node not being found in the partition, things worked.

David

Great! I'm changing this to a sev-3.

Two questions:

* When did you upgrade to 18.08.8?
* When was the offending job (3222565) submitted?
(sacct -j 3222565 --format=jobid,submit)

18.08.8 had a fix (commit 4c48a84a6edb) to prevent corrupted job structures, but if the job was submitted before 18.08.8 then it's possible the job struct was corrupted. If the job was submitted after 18.08.8, then we need to fix something else.

Marshall,

Thanks. I'm working to track down when we went to 18.08.8, but I don't think it was within the past couple weeks. The job info:

```
[root@glctld ~]# sacct -j 3222565 --format=jobid,submit
       JobID              Submit
------------ -------------------
3222565      2020-01-10T12:34:33
3222565.ext+ 2020-01-10T12:36:57
```

David

Marhsall,

From our git commits, it looks like we did the move to 18.08.8 in late September early October.

Best,

David

Thanks for the additional information. I'll keep looking into this. Can you keep the core file and slurmctld binary in case I ask for additional gdb output?

Absolutely!

There were a few core dumps as we tried to restart things to understand them better. So, there's a treasure trove of stuff to be mined if needed.

Thanks again!!

David

This can be closed.
We found a node in slurm.conf that was removed from the cluster.  Once slurm.conf was updated by removing that node all things started to work.
We have also now upgraded to 19.05.5

Thanks
Vasile

We discovered the issue to the problem.