Slurm version(s) affected: 22.05 through 25.11 (all maintained stable branches). Fixed on master (26.11) by refactor commit `710f909b0c`. --- ## Summary `slurmctld` reliably segfaults when it receives a `REQUEST_ASSOC_MGR_INFO` RPC (e.g. `scontrol show assoc_mgr`) while the association or QOS caches are NULL. This happens whenever the assoc_mgr lists have not been populated — most commonly with `AccountingStorageType=accounting_storage/none`, or during the window where slurmctld has started but has not yet connected to slurmdbd. The USERS section of `assoc_mgr_info_get_pack_msg()` already guards against a NULL list, but the ASSOC and QOS sections do not, so they call `list_iterator_create()` on a NULL pointer. ## Steps to reproduce 1. Configure a controller with `AccountingStorageType=accounting_storage/none` (or start slurmctld before slurmdbd is reachable). 2. Run `scontrol show assoc_mgr`. 3. `slurmctld` crashes with SIGSEGV. With no arguments, `scontrol show assoc_mgr` sets `req.flags = ASSOC_MGR_INFO_FLAG_ASSOC | ASSOC_MGR_INFO_FLAG_USERS | ASSOC_MGR_INFO_FLAG_QOS` (`src/scontrol/info_assoc_mgr.c`), so the unguarded ASSOC path is always taken. ## Backtrace Observed on 23.11.11 (Ubuntu 24.04 package, `-DNDEBUG` build): ``` #0 __pthread_rwlock_wrlock (rwlock=0x30) at pthread_rwlock_common.c #2 list_iterator_create (l=0x30) at src/common/list.c:823 #3 assoc_mgr_info_get_pack_msg (..., uid=..., protocol_version=10240) at src/common/assoc_mgr.c:3614 #4 _slurm_rpc_assoc_mgr_info (msg=...) at src/slurmctld/proc_req.c:5994 #5 slurmctld_req (msg=...) at src/slurmctld/proc_req.c:6975 #6 _service_connection (...) at src/slurmctld/controller.c:1477 ``` ## Root cause `assoc_mgr_assoc_list` is `NULL` at init and only assigned a real list after `_get_assoc_mgr_assoc_list()` succeeds (or via an assoc update RPC). In `assoc_mgr_info_get_pack_msg()` the three sections are inconsistent: - **ASSOC** (23.11 line 3614): `if (!(flags & ASSOC_MGR_INFO_FLAG_ASSOC)) goto no_assocs;` — no NULL check, then `list_iterator_create(assoc_mgr_assoc_list)` at line 3617. - **QOS** (line 3693): `if (!(flags & ASSOC_MGR_INFO_FLAG_QOS)) { ... }` — no NULL check. - **USERS** (line 3725): `if (!(flags & ASSOC_MGR_INFO_FLAG_USERS) || !assoc_mgr_user_list) goto no_users;` — **already guarded**. In a release build `xassert(l != NULL)` in `list_iterator_create()` is compiled out (`xassert.h`: `#ifdef NDEBUG #define xassert(expr) ((void)(0))`), so execution reaches `slurm_rwlock_wrlock(&l->mutex)` with `l == NULL`. `mutex` is at offset `0x30` in `struct xlist`, giving the `rwlock=0x30` fault address seen in the backtrace. The same unguarded pattern is present in all maintained stable branches (verified by reading `src/common/assoc_mgr.c` on 22.05.11, 23.02.8, 23.11.11, 24.05.8, 24.11.7, 25.05.9, 25.11.8). Line numbers of the two unguarded checks: | Branch | ASSOC check | QOS check | |--------|-------------|-----------| | 23.11 | 3614 | 3693 | | 25.11 | 3743 | 3822 | master (26.11) is **not** affected: `710f909b0c` ("Pack inline in assoc_mgr_info_get_pack_msg() and drop ret_list") reworked the function into `_pack_filtered_section()`, which does `if (list) list_for_each_ro(...)`. ## Suggested minimal fix (for stable branches) Mirror the existing USERS guard on the ASSOC and QOS sections: ```diff - if (!(flags & ASSOC_MGR_INFO_FLAG_ASSOC)) + if (!(flags & ASSOC_MGR_INFO_FLAG_ASSOC) || !assoc_mgr_assoc_list) goto no_assocs; ... - if (!(flags & ASSOC_MGR_INFO_FLAG_QOS)) { + if (!(flags & ASSOC_MGR_INFO_FLAG_QOS) || !assoc_mgr_qos_list) { tmp_list = ret_list; goto no_qos; } ``` This makes the ASSOC/QOS paths behave like the USERS path (return an empty section instead of crashing). It is a good backport candidate for the stable branches, since the full master refactor is too invasive for a stable/distro update.