Ticket 25708 - slurmctld segfault in assoc_mgr_info_get_pack_msg() on NULL assoc/qos list (scontrol show assoc_mgr without slurmdbd data)
Summary: slurmctld segfault in assoc_mgr_info_get_pack_msg() on NULL assoc/qos list (s...
Status: OPEN
Alias: None
Product: Slurm
Classification: Unclassified
Component: slurmctld (show other tickets)
Version: 25.11.7
Hardware: Linux Linux
: 6 - No support contract
Assignee: Jacob Jenson
QA Contact:
URL:
Depends on:
Blocks:
 
Reported: 2026-08-13 23:15 MDT by Shengqi Chen
Modified: 2026-08-13 23:15 MDT (History)
1 user (show)

See Also:
Site: -Other-
Slinky Site: ---
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
Google sites: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
NoveTech Sites: ---
Nvidia HWinf-CS Sites: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Tzag Elita Sites: ---
Linux Distro: Ubuntu
Machine Name:
CLE Version:
Version Fixed:
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---


Attachments

Note You need to log in before you can comment on or make changes to this ticket.
Description Shengqi Chen 2026-08-13 23:15:21 MDT
Slurm version(s) affected: 22.05 through 25.11 (all maintained stable branches). Fixed on master (26.11) by refactor commit `710f909b0c`.

---

## Summary

`slurmctld` reliably segfaults when it receives a `REQUEST_ASSOC_MGR_INFO` RPC (e.g. `scontrol show assoc_mgr`) while the association or QOS caches are NULL. This happens whenever the assoc_mgr lists have not been populated — most commonly with `AccountingStorageType=accounting_storage/none`, or during the window where slurmctld has started but has not yet connected to slurmdbd.

The USERS section of `assoc_mgr_info_get_pack_msg()` already guards against a NULL list, but the ASSOC and QOS sections do not, so they call `list_iterator_create()` on a NULL pointer.

## Steps to reproduce

1. Configure a controller with `AccountingStorageType=accounting_storage/none` (or start slurmctld before slurmdbd is reachable).
2. Run `scontrol show assoc_mgr`.
3. `slurmctld` crashes with SIGSEGV.

With no arguments, `scontrol show assoc_mgr` sets `req.flags = ASSOC_MGR_INFO_FLAG_ASSOC | ASSOC_MGR_INFO_FLAG_USERS | ASSOC_MGR_INFO_FLAG_QOS` (`src/scontrol/info_assoc_mgr.c`), so the unguarded ASSOC path is always taken.

## Backtrace

Observed on 23.11.11 (Ubuntu 24.04 package, `-DNDEBUG` build):

```
#0  __pthread_rwlock_wrlock (rwlock=0x30) at pthread_rwlock_common.c
#2  list_iterator_create (l=0x30) at src/common/list.c:823
#3  assoc_mgr_info_get_pack_msg (..., uid=..., protocol_version=10240) at src/common/assoc_mgr.c:3614
#4  _slurm_rpc_assoc_mgr_info (msg=...) at src/slurmctld/proc_req.c:5994
#5  slurmctld_req (msg=...) at src/slurmctld/proc_req.c:6975
#6  _service_connection (...) at src/slurmctld/controller.c:1477
```

## Root cause

`assoc_mgr_assoc_list` is `NULL` at init and only assigned a real list after `_get_assoc_mgr_assoc_list()` succeeds (or via an assoc update RPC). In `assoc_mgr_info_get_pack_msg()` the three sections are inconsistent:

- **ASSOC** (23.11 line 3614): `if (!(flags & ASSOC_MGR_INFO_FLAG_ASSOC)) goto no_assocs;` — no NULL check, then `list_iterator_create(assoc_mgr_assoc_list)` at line 3617.
- **QOS** (line 3693): `if (!(flags & ASSOC_MGR_INFO_FLAG_QOS)) { ... }` — no NULL check.
- **USERS** (line 3725): `if (!(flags & ASSOC_MGR_INFO_FLAG_USERS) || !assoc_mgr_user_list) goto no_users;` — **already guarded**.

In a release build `xassert(l != NULL)` in `list_iterator_create()` is compiled out (`xassert.h`: `#ifdef NDEBUG #define xassert(expr) ((void)(0))`), so execution reaches `slurm_rwlock_wrlock(&l->mutex)` with `l == NULL`. `mutex` is at offset `0x30` in `struct xlist`, giving the `rwlock=0x30` fault address seen in the backtrace.

The same unguarded pattern is present in all maintained stable branches (verified by reading `src/common/assoc_mgr.c` on 22.05.11, 23.02.8, 23.11.11, 24.05.8, 24.11.7, 25.05.9, 25.11.8). Line numbers of the two unguarded checks:

| Branch | ASSOC check | QOS check |
|--------|-------------|-----------|
| 23.11  | 3614        | 3693      |
| 25.11  | 3743        | 3822      |

master (26.11) is **not** affected: `710f909b0c` ("Pack inline in assoc_mgr_info_get_pack_msg() and drop ret_list") reworked the function into `_pack_filtered_section()`, which does `if (list) list_for_each_ro(...)`.

## Suggested minimal fix (for stable branches)

Mirror the existing USERS guard on the ASSOC and QOS sections:

```diff
-	if (!(flags & ASSOC_MGR_INFO_FLAG_ASSOC))
+	if (!(flags & ASSOC_MGR_INFO_FLAG_ASSOC) || !assoc_mgr_assoc_list)
 		goto no_assocs;
...
-	if (!(flags & ASSOC_MGR_INFO_FLAG_QOS)) {
+	if (!(flags & ASSOC_MGR_INFO_FLAG_QOS) || !assoc_mgr_qos_list) {
 		tmp_list = ret_list;
 		goto no_qos;
 	}
```

This makes the ASSOC/QOS paths behave like the USERS path (return an empty section instead of crashing). It is a good backport candidate for the stable branches, since the full master refactor is too invasive for a stable/distro update.