Created attachment 46015 [details] Graph showing memory usage of slurmctld over time. I'm seeing unexplained memory growth for slurmctld after we upgraded to 26.05.2. It was fine for a while but since about yesterday at 4pm EDT it has started growing steadily. Restarting slurmctld seems to reset it but this indicates a memory leak somewhere. I've attached a graph. Let me know if there is any data you want me to pull (including if you want me to force a core dump).
FYI, I'm going to add a cronjob to our slurm master to restart slurmctld every 6 hours to head off this problem.
>I'm seeing unexplained memory growth for slurmctld after we upgraded to 26.05.2... >FYI, I'm going to add a cronjob to our slurm master to restart slurmctld every 6 hours to head off this problem. Thanks for logging this. We have a few that we have patched landing in 26.05.3 with a tentative release this Thursday. By chance does this one fit? Do you have an idea what is triggering it? > slurmctld - Fix memory leak of a multi-QOS list on a rejected job update
Not obviously. I did just log this: https://support.schedmd.com/show_bug.cgi?id=25687 But it was more informational. That said I do know that the jobs in the kempner queues tend to interact with multiple QoS's. So if that is the cause of the bug, then yeah we are definitely going to hit that one and it is possibly the cause. -Paul Edmon- On 8/11/26 2:29 PM, bugs@schedmd.com wrote: > > *Comment # 2 <https://support.schedmd.com/show_bug.cgi?id=25685#c2> on > ticket 25685 <https://support.schedmd.com/show_bug.cgi?id=25685> from > Jason Booth <mailto:jbooth@nvidia.com> * > >I'm seeing unexplained memory growth for slurmctld after we upgraded to 26.05.2... >FYI, I'm going to add a cronjob to our slurm master to restart > slurmctld every 6 hours to head off this problem. > > Thanks for logging this. We have a few that we have patched landing in 26.05.3 > with a tentative release this Thursday. > > By chance does this one fit? Do you have an idea what is triggering it? > > > slurmctld - Fix memory leak of a multi-QOS list on a rejected job update > ------------------------------------------------------------------------ > You are receiving this mail because: > > * You reported the ticket. >
Just took a look and the existence of the gres error in 25687 does not coincide with the memory leak. So its not that specifically. That said it could still be the QoS thing. It depends though on the details of the cause of that leak. (In reply to Paul Edmon from comment #3) > Not obviously. I did just log this: > https://support.schedmd.com/show_bug.cgi?id=25687 But it was more > informational. That said I do know that the jobs in the kempner queues > tend to interact with multiple QoS's. So if that is the cause of the > bug, then yeah we are definitely going to hit that one and it is > possibly the cause. > > -Paul Edmon- > > On 8/11/26 2:29 PM, bugs@schedmd.com wrote: > > > > *Comment # 2 <https://support.schedmd.com/show_bug.cgi?id=25685#c2> on > > ticket 25685 <https://support.schedmd.com/show_bug.cgi?id=25685> from > > Jason Booth <mailto:jbooth@nvidia.com> * > > >I'm seeing unexplained memory growth for slurmctld after we upgraded to 26.05.2... >FYI, I'm going to add a cronjob to our slurm master to restart > > slurmctld every 6 hours to head off this problem. > > > > Thanks for logging this. We have a few that we have patched landing in 26.05.3 > > with a tentative release this Thursday. > > > > By chance does this one fit? Do you have an idea what is triggering it? > > > > > slurmctld - Fix memory leak of a multi-QOS list on a rejected job update > > ------------------------------------------------------------------------ > > You are receiving this mail because: > > > > * You reported the ticket. > >
Hi Paul, Between 26.05.2 and 26.05.3, only two leak fixes landed, and both are slurmctld-specific: - a56e01f - slurmctld: Fix memory leak of a multi-reservation list on job update - 7a6f18e - slurmctld: Fix memory leak of a multi-QOS list on a rejected job update Both fixes are in _update_job() in src/slurmctld/job_mgr.c, and both only leak on rejected/failed job updates: 1. a56e01f - a scontrol update job Reservation=a,b (multi-reservation) request that fails validation leaked the new_resv_list. 2. 7a6f18e - a scontrol update job QOS=a,b (multi-QOS) request that fails validation leaked the new_qos_list. These leaks only fire when a job update to a running job requesting multiple comma-separated reservations or QOS values is rejected by validation. If the leak correlates with scontrol update job calls specifying multi-value Reservation= or QOS= on jobs (especially update attempts that fail, e.g. invalid combos, jobs in the wrong state, etc.), this is very likely your bug and 26.05.3 should fix it. If your leak isn't tied to job-update RPCs at all (e.g. it grows purely from job submission/scheduling churn with no scontrol update traffic), it's probably a different, still-unfixed leak. Questions: 1. Do "scontrol update" calls fail? 2. Do you have a test environment where you can evaluate 26.05.3? 3. What Slurm version did you run prior to 26.05.2? 4. Can you provide a list of the commands you run so I can try to reproduce it this leak? Ref: https://github.com/SchedMD/slurm/commit/a56e01f6df43551efc52eb3ebe258d59ddb8d5bb https://github.com/SchedMD/slurm/commit/7a6f18e6edaf9998968849d126da45962e494719 -Sheldon
Great questions. I think my plan of action is to upgrade to 26.05.3 at the soonest opportunity just to see if it fixes these. This leak is only happening on our main scheduler and not on our test environment so its pretty clearly something in that setup. We do have multiple overlapping QoS's between partition QoS's and account QoS's so I suspect that is the likely cause. That said won't know for certain until after I get this upgrade done, which I am hoping to do in the next day or two. Once it is done and I confirm whether the memory leak is still happening I will let you know along with the answers to these questions. -Paul Edmon- On 8/14/2026 12:08 PM, bugs@schedmd.com wrote: > > *Comment # 5 <https://support.schedmd.com/show_bug.cgi?id=25685#c5> on > ticket 25685 <https://support.schedmd.com/show_bug.cgi?id=25685> from > Sheldon Teerlink <mailto:steerlink@nvidia.com> * > Hi Paul, > > Between 26.05.2 and 26.05.3, only two leak fixes landed, and both are > slurmctld-specific: > > - a56e01f - slurmctld: Fix memory leak of a multi-reservation list on job > update > - 7a6f18e - slurmctld: Fix memory leak of a multi-QOS list on a rejected job > update > > Both fixes are in _update_job() in src/slurmctld/job_mgr.c, and both only leak > on rejected/failed job updates: > > 1. a56e01f - a scontrol update job Reservation=a,b (multi-reservation) request > that fails validation leaked the new_resv_list. > 2. 7a6f18e - a scontrol update job QOS=a,b (multi-QOS) request that fails > validation leaked the new_qos_list. > > These leaks only fire when a job update to a running job requesting multiple > comma-separated reservations or QOS values is rejected by validation. If the > leak correlates with scontrol update job calls specifying multi-value > Reservation= or QOS= on jobs (especially update attempts that fail, e.g. > invalid combos, jobs in the wrong state, etc.), this is very likely your bug > and 26.05.3 should fix it. > > If your leak isn't tied to job-update RPCs at all (e.g. it grows purely from > job submission/scheduling churn with no scontrol update traffic), it's probably > a different, still-unfixed leak. > > > Questions: > 1. Do "scontrol update" calls fail? > 2. Do you have a test environment where you can evaluate 26.05.3? > 3. What Slurm version did you run prior to 26.05.2? > 4. Can you provide a list of the commands you run so I can try to reproduce it > this leak? > > > Ref: > > https://github.com/SchedMD/slurm/commit/a56e01f6df43551efc52eb3ebe258d59ddb8d5bb > https://github.com/SchedMD/slurm/commit/7a6f18e6edaf9998968849d126da45962e494719 > > -Sheldon > ------------------------------------------------------------------------ > You are receiving this mail because: > > * You reported the ticket. >
