Created attachment 46015 [details] Graph showing memory usage of slurmctld over time. I'm seeing unexplained memory growth for slurmctld after we upgraded to 26.05.2. It was fine for a while but since about yesterday at 4pm EDT it has started growing steadily. Restarting slurmctld seems to reset it but this indicates a memory leak somewhere. I've attached a graph. Let me know if there is any data you want me to pull (including if you want me to force a core dump).
FYI, I'm going to add a cronjob to our slurm master to restart slurmctld every 6 hours to head off this problem.
>I'm seeing unexplained memory growth for slurmctld after we upgraded to 26.05.2... >FYI, I'm going to add a cronjob to our slurm master to restart slurmctld every 6 hours to head off this problem. Thanks for logging this. We have a few that we have patched landing in 26.05.3 with a tentative release this Thursday. By chance does this one fit? Do you have an idea what is triggering it? > slurmctld - Fix memory leak of a multi-QOS list on a rejected job update
Not obviously. I did just log this: https://support.schedmd.com/show_bug.cgi?id=25687 But it was more informational. That said I do know that the jobs in the kempner queues tend to interact with multiple QoS's. So if that is the cause of the bug, then yeah we are definitely going to hit that one and it is possibly the cause. -Paul Edmon- On 8/11/26 2:29 PM, bugs@schedmd.com wrote: > > *Comment # 2 <https://support.schedmd.com/show_bug.cgi?id=25685#c2> on > ticket 25685 <https://support.schedmd.com/show_bug.cgi?id=25685> from > Jason Booth <mailto:jbooth@nvidia.com> * > >I'm seeing unexplained memory growth for slurmctld after we upgraded to 26.05.2... >FYI, I'm going to add a cronjob to our slurm master to restart > slurmctld every 6 hours to head off this problem. > > Thanks for logging this. We have a few that we have patched landing in 26.05.3 > with a tentative release this Thursday. > > By chance does this one fit? Do you have an idea what is triggering it? > > > slurmctld - Fix memory leak of a multi-QOS list on a rejected job update > ------------------------------------------------------------------------ > You are receiving this mail because: > > * You reported the ticket. >
Just took a look and the existence of the gres error in 25687 does not coincide with the memory leak. So its not that specifically. That said it could still be the QoS thing. It depends though on the details of the cause of that leak. (In reply to Paul Edmon from comment #3) > Not obviously. I did just log this: > https://support.schedmd.com/show_bug.cgi?id=25687 But it was more > informational. That said I do know that the jobs in the kempner queues > tend to interact with multiple QoS's. So if that is the cause of the > bug, then yeah we are definitely going to hit that one and it is > possibly the cause. > > -Paul Edmon- > > On 8/11/26 2:29 PM, bugs@schedmd.com wrote: > > > > *Comment # 2 <https://support.schedmd.com/show_bug.cgi?id=25685#c2> on > > ticket 25685 <https://support.schedmd.com/show_bug.cgi?id=25685> from > > Jason Booth <mailto:jbooth@nvidia.com> * > > >I'm seeing unexplained memory growth for slurmctld after we upgraded to 26.05.2... >FYI, I'm going to add a cronjob to our slurm master to restart > > slurmctld every 6 hours to head off this problem. > > > > Thanks for logging this. We have a few that we have patched landing in 26.05.3 > > with a tentative release this Thursday. > > > > By chance does this one fit? Do you have an idea what is triggering it? > > > > > slurmctld - Fix memory leak of a multi-QOS list on a rejected job update > > ------------------------------------------------------------------------ > > You are receiving this mail because: > > > > * You reported the ticket. > >
Hi Paul, Between 26.05.2 and 26.05.3, only two leak fixes landed, and both are slurmctld-specific: - a56e01f - slurmctld: Fix memory leak of a multi-reservation list on job update - 7a6f18e - slurmctld: Fix memory leak of a multi-QOS list on a rejected job update Both fixes are in _update_job() in src/slurmctld/job_mgr.c, and both only leak on rejected/failed job updates: 1. a56e01f - a scontrol update job Reservation=a,b (multi-reservation) request that fails validation leaked the new_resv_list. 2. 7a6f18e - a scontrol update job QOS=a,b (multi-QOS) request that fails validation leaked the new_qos_list. These leaks only fire when a job update to a running job requesting multiple comma-separated reservations or QOS values is rejected by validation. If the leak correlates with scontrol update job calls specifying multi-value Reservation= or QOS= on jobs (especially update attempts that fail, e.g. invalid combos, jobs in the wrong state, etc.), this is very likely your bug and 26.05.3 should fix it. If your leak isn't tied to job-update RPCs at all (e.g. it grows purely from job submission/scheduling churn with no scontrol update traffic), it's probably a different, still-unfixed leak. Questions: 1. Do "scontrol update" calls fail? 2. Do you have a test environment where you can evaluate 26.05.3? 3. What Slurm version did you run prior to 26.05.2? 4. Can you provide a list of the commands you run so I can try to reproduce it this leak? Ref: https://github.com/SchedMD/slurm/commit/a56e01f6df43551efc52eb3ebe258d59ddb8d5bb https://github.com/SchedMD/slurm/commit/7a6f18e6edaf9998968849d126da45962e494719 -Sheldon
Great questions. I think my plan of action is to upgrade to 26.05.3 at the soonest opportunity just to see if it fixes these. This leak is only happening on our main scheduler and not on our test environment so its pretty clearly something in that setup. We do have multiple overlapping QoS's between partition QoS's and account QoS's so I suspect that is the likely cause. That said won't know for certain until after I get this upgrade done, which I am hoping to do in the next day or two. Once it is done and I confirm whether the memory leak is still happening I will let you know along with the answers to these questions. -Paul Edmon- On 8/14/2026 12:08 PM, bugs@schedmd.com wrote: > > *Comment # 5 <https://support.schedmd.com/show_bug.cgi?id=25685#c5> on > ticket 25685 <https://support.schedmd.com/show_bug.cgi?id=25685> from > Sheldon Teerlink <mailto:steerlink@nvidia.com> * > Hi Paul, > > Between 26.05.2 and 26.05.3, only two leak fixes landed, and both are > slurmctld-specific: > > - a56e01f - slurmctld: Fix memory leak of a multi-reservation list on job > update > - 7a6f18e - slurmctld: Fix memory leak of a multi-QOS list on a rejected job > update > > Both fixes are in _update_job() in src/slurmctld/job_mgr.c, and both only leak > on rejected/failed job updates: > > 1. a56e01f - a scontrol update job Reservation=a,b (multi-reservation) request > that fails validation leaked the new_resv_list. > 2. 7a6f18e - a scontrol update job QOS=a,b (multi-QOS) request that fails > validation leaked the new_qos_list. > > These leaks only fire when a job update to a running job requesting multiple > comma-separated reservations or QOS values is rejected by validation. If the > leak correlates with scontrol update job calls specifying multi-value > Reservation= or QOS= on jobs (especially update attempts that fail, e.g. > invalid combos, jobs in the wrong state, etc.), this is very likely your bug > and 26.05.3 should fix it. > > If your leak isn't tied to job-update RPCs at all (e.g. it grows purely from > job submission/scheduling churn with no scontrol update traffic), it's probably > a different, still-unfixed leak. > > > Questions: > 1. Do "scontrol update" calls fail? > 2. Do you have a test environment where you can evaluate 26.05.3? > 3. What Slurm version did you run prior to 26.05.2? > 4. Can you provide a list of the commands you run so I can try to reproduce it > this leak? > > > Ref: > > https://github.com/SchedMD/slurm/commit/a56e01f6df43551efc52eb3ebe258d59ddb8d5bb > https://github.com/SchedMD/slurm/commit/7a6f18e6edaf9998968849d126da45962e494719 > > -Sheldon > ------------------------------------------------------------------------ > You are receiving this mail because: > > * You reported the ticket. >
We upgraded to Slurm 26.05.3 yesterday and I'm still seeing concerning memory growth for slurmctld. I've attached a graph since I restarted it this morning. In the previous version we were running 25.11.5 slurmctld capped out at 12 GB. In this new version it is now up to 40GB after 8 hours and I don't see it plateauing yet. I think the memory leak may still exist (I'm not 100% yet as I'm not sure if it will plateau at some point or if it will keep growing slowly). As to your questions: 1. No they work fine. I've run several to close nodes or update job priority or partitions. 2. Yes and no. We have a test cluster but it is not showing the growth in memory like this is. This appears to be something that is unique to our main cluster Cannon. For example our other cluster FASSE running the same version of Slurm but at much smaller scale is only using 0.5 GB of memory for slurmctld. 3. 25.11.5 4. Sadly no, I have no idea what is causing the memory leak or anomalous growth. I guess the question is, do you have a heuristic as to how much memory slurmctld should ordinarily used based on job count and cluster size? If so we can then derive what it should be versus what it is. Given that the previous version only used 12 GB, I'm dubious that the memory usage would have changed that much unless you dramatically overhauled the internal memory structure of the slurmctld. If that is the case then we should derive what the theoretical peak memory would be and then see if we are out of range. I'm also happy to run diagnostics and take coredumps of whatever you need. In the meantime I'm going to keep an eye on this to see if it keeps growing even larger and if so I will probably have to reintroduce my forced restarts of slurmctld every 6-12 hours to keep things under control.
Created attachment 46064 [details] Graph showing memory usage for Slurm 26.05.3 on August 19 2026
I'm working on some instructions you can follow to collect memory data from the leaking slurmctld process in production. In the meantime, can you answer the following questions? 1. What process do you use to build slurmctld? (e.g. ./configure, RPM, etc.). If you're using the basic ./configure approach, what args do you use? 2. What is the output of the following command? Modify it as necessary depending on your path. Ultimately I need to know if the slurmctld binary has symbols or whether you can build it with symbols (non stripped). > file $(command -v slurmctld) 3. Are you running slurmctld in HA mode with primary and backup controllers. -Sheldon
Looks like we stalled out at around 41 GB. Or at least we have been holding even at that for the past 4 hours. In answer to your questions. 1. rpmbuild. You can see our spec here: https://github.com/fasrc/slurm-spec/tree/26.05.3-1fasrc01 For reference we are running Rocky 8.10 2. [root@holy-slurm02 ~]# file $(command -v slurmctld) /usr/sbin/slurmctld: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=cafb8c480ae28790ed36beb2e80e8fe05aa5ab2f, with debug_info, not stripped, too many notes (256) 3. We are not running in HA mode. We have a single controller with no backup for failover. -Paul Edmon- On 8/19/2026 6:15 PM, bugs@schedmd.com wrote: > > *Comment # 9 <https://support.schedmd.com/show_bug.cgi?id=25685#c9> on > ticket 25685 <https://support.schedmd.com/show_bug.cgi?id=25685> from > Sheldon Teerlink <mailto:steerlink@nvidia.com> * > I'm working on some instructions you can follow to collect memory data from the > leaking slurmctld process in production. In the meantime, can you answer the > following questions? > > 1. What process do you use to build slurmctld? (e.g. ./configure, RPM, etc.). > If you're using the basic ./configure approach, what args do you use? > 2. What is the output of the following command? Modify it as necessary > depending on your path. Ultimately I need to know if the slurmctld binary has > symbols or whether you can build it with symbols (non stripped). > > > file $(command -v slurmctld) > > 3. Are you running slurmctld in HA mode with primary and backup controllers. > > -Sheldon > ------------------------------------------------------------------------ > You are receiving this mail because: > > * You reported the ticket. >
Also just for reference: [root@holy-slurm02 ~]# ps aux | grep "/usr/sbin/slurmctld" root 1497987 0.0 0.0 12216 1272 pts/0 S+ 20:13 0:00 grep --color=auto /usr/sbin/slurmctld slurm 3969983 145 20.6 60063292 40605792 ? SLsl 06:16 1213:58 /usr/sbin/slurmctld --systemd It's been holding at about that for the past 4 hours or so.
Just an update. The memory usage held firm at 40GB all night. At this point I'm pretty confident that the memory leak is fixed, but it would probably be good to understand why slurmctld now requires triple the amount of memory in 26.05 from 25.11.
Hi Paul, Thanks for the update. 40GB RSS seems high to me. Can you please check your MinJobAge value in slurm.conf and let me know what it is? A high value keeps completed jobs in memory for a while. Also, can you determine how deep the job queue is, as that will occupy memory? > squeue -h -t all | wc -l While it's good to know the memory plateaued, there could still be a leak. The following are instructions on how to profile slurmctld and capture periodic heap profiles using tcmalloc. If you're able to do this and provide the analysis between heap dumps while memory is growing that would really help us locate the memory growth. The slurmctld you're running has symbols so we'll get the information we need. Heap Profiling slurmctld in Production with tcmalloc (gperftools) ==================================================================== Works via LD_PRELOAD against the existing slurmctld binary -- no rebuild or instrumented build required. The general idea here is to track all memory allocations made by slurmctld and then analyze them over time. This requires restarting the slurmctld process as described in this document. Adjust the paths and values as needed. 1. Install ---------- RHEL/Rocky: dnf install gperftools gperftools-libs gperftools-devel Debian/Ubuntu: apt install google-perftools libgoogle-perftools-dev Confirm the library is present: find / -name 'libtcmalloc_and_profiler.so*' 2>/dev/null Use libtcmalloc_and_profiler.so (not plain libtcmalloc.so) -- it bundles the heap profiler. 2. Wire it into the slurmctld systemd unit via a drop-in ---------------------------------------------------------- Do not edit the shipped unit file directly. Create an override instead ensuring the correct paths are used: mkdir -p /etc/systemd/system/slurmctld.service.d cat > /etc/systemd/system/slurmctld.service.d/heapprofile.conf <<'EOF' [Service] Environment=LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libtcmalloc_and_profiler.so.4 Environment=HEAPPROFILE=/var/spool/slurmctld/heapprof/slurmctld.hprof Environment=HEAP_PROFILE_TIME_INTERVAL=3600 Environment=HEAP_PROFILE_ALLOCATION_INTERVAL=1073741824 EOF mkdir -p /var/spool/slurmctld/heapprof systemctl daemon-reload Key env vars: - HEAP_PROFILE_TIME_INTERVAL=3600 Forces a dump every hour regardless of allocation volume. This is the one that matters for a slow leak -- without it, dumps only fire after HEAP_PROFILE_ALLOCATION_INTERVAL bytes (default 1GB) have been allocated in total, which may take a long time if the leak is small relative to normal allocation churn. - HEAP_PROFILE_ALLOCATION_INTERVAL Leave at default (1GB) or lower for more frequent dumps early on. Each dump is small (KB to low MB), so err toward more frequent. - Optional: HEAP_PROFILE_INUSE_INTERVAL to also dump on new in-use memory peaks. 3. Restart under HA -------------------- LD_PRELOAD requires a process restart -- cannot be applied with `scontrol reconfigure`. If running primary/backup (SlurmctldHost[0]/SlurmctldHost[1]): 1. Apply the drop-in to the BACKUP controller first, restart it, confirm `sdiag` / `scontrol ping` look healthy and it is acting as a normal backup. 2. Manually fail over (`scontrol takeover`) or wait for a maintenance window, then apply to primary and restart it. 3. This gets profiling running on whichever instance is acting primary with only the brief unavailability normal to any slurmctld restart. If single-controller, this needs a scheduled maintenance window -- same downtime as any ordinary slurmctld restart/upgrade, nothing extra caused by the profiler itself. 4. Let it run -------------- Let it accumulate dumps for as long as it takes to see meaningful RSS growth -- hours to a few days depending on leak rate. Files land as: /var/spool/slurmctld/heapprof/slurmctld.hprof.0001.heap /var/spool/slurmctld/heapprof/slurmctld.hprof.0002.heap ... Watch disk usage; each file is small, but keep an eye on the spool directory. 5. Analyze ----------- Copy a few dumps spanning the growth window to an analysis host along with the exact slurmctld binary you're are running (with symbols -- if stripped, grab the matching -debuginfo/-dbgsym package for that build; no rebuild needed). # Cumulative in-use memory at a single point, top offenders: pprof --text /usr/sbin/slurmctld slurmctld.hprof.0010.heap # THE useful one -- diff an early snapshot against a late snapshot # to isolate what grew between the two, filtering out steady-state # background usage: google-pprof --base=slurmctld.hprof.0002.heap \ /usr/sbin/slurmctld slurmctld.hprof.0020.heap --text # Visual call graph of the diff (needs graphviz): google-pprof --base=slurmctld.hprof.0002.heap \ /usr/sbin/slurmctld slurmctld.hprof.0020.heap --pdf > growth.pdf The --base diff is the payoff -- it shows exactly which call stacks accumulated allocations between the two snapshots, in bytes and object count, sorted descending. That is the leak site, directly. 6. Revert when done --------------------- rm /etc/systemd/system/slurmctld.service.d/heapprofile.conf systemctl daemon-reload systemctl restart slurmctld # backup first, then failover back if desired Overhead / safety notes ------------------------ - Adds roughly 5-15% CPU/memory bookkeeping overhead at these settings -- much lighter than valgrind (10-50x) and safe for a live scheduling controller, but still worth doing on the backup first / during a lower-traffic period if there is any concern. - No changes to the Slurm build -- same RPM/binary, just LD_PRELOAD + env vars via the systemd drop-in. - If the controller is memory-constrained already, the profiler's own bookkeeping adds some overhead on top of RSS -- usually not significant, but worth noting if you are near a memory ceiling. =============================================================================== -Sheldon
[root@holy-slurm02 ~]# scontrol show conf | grep MinJobAge MinJobAge = 600 sec [root@holy-slurm02 ~]# squeue -h -t all | wc -l 36057 By the way that is a pretty typical queue length for us. As for the profiling, I will set that up tomorrow and run it during the day while I am able to watch it. I will let you know what I find. -Paul Edmon- On 8/20/26 3:26 PM, bugs@schedmd.com wrote: > > *Comment # 13 <https://support.schedmd.com/show_bug.cgi?id=25685#c13> > on ticket 25685 <https://support.schedmd.com/show_bug.cgi?id=25685> > from Sheldon Teerlink <mailto:steerlink@nvidia.com> * > Hi Paul, > > Thanks for the update. 40GB RSS seems high to me. Can you please check your > MinJobAge value in slurm.conf and let me know what it is? A high value keeps > completed jobs in memory for a while. Also, can you determine how deep the job > queue is, as that will occupy memory? > > > squeue -h -t all | wc -l > > While it's good to know the memory plateaued, there could still be a leak. The > following are instructions on how to profile slurmctld and capture periodic > heap profiles using tcmalloc. If you're able to do this and provide the > analysis between heap dumps while memory is growing that would really help us > locate the memory growth. > > The slurmctld you're running has symbols so we'll get the information we need. > > > Heap Profiling slurmctld in Production with tcmalloc (gperftools) > ==================================================================== > > Works via LD_PRELOAD against the existing slurmctld binary -- no rebuild > or instrumented build required. > > The general idea here is to track all memory allocations made by > slurmctld and then analyze them over time. This requires restarting > the slurmctld process as described in this document. > > Adjust the paths and values as needed. > > 1. Install > ---------- > RHEL/Rocky: > dnf install gperftools gperftools-libs gperftools-devel > > Debian/Ubuntu: > apt install google-perftools libgoogle-perftools-dev > > Confirm the library is present: > find / -name 'libtcmalloc_and_profiler.so*' 2>/dev/null > > Use libtcmalloc_and_profiler.so (not plain libtcmalloc.so) -- it bundles > the heap profiler. > > 2. Wire it into the slurmctld systemd unit via a drop-in > ---------------------------------------------------------- > Do not edit the shipped unit file directly. Create an override instead > ensuring the correct paths are used: > > mkdir -p /etc/systemd/system/slurmctld.service.d > cat > /etc/systemd/system/slurmctld.service.d/heapprofile.conf <<'EOF' > [Service] > > Environment=LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libtcmalloc_and_profiler.so.4 > Environment=HEAPPROFILE=/var/spool/slurmctld/heapprof/slurmctld.hprof > Environment=HEAP_PROFILE_TIME_INTERVAL=3600 > Environment=HEAP_PROFILE_ALLOCATION_INTERVAL=1073741824 > EOF > mkdir -p /var/spool/slurmctld/heapprof > systemctl daemon-reload > > Key env vars: > - HEAP_PROFILE_TIME_INTERVAL=3600 > Forces a dump every hour regardless of allocation volume. This is > the one that matters for a slow leak -- without it, dumps only > fire after HEAP_PROFILE_ALLOCATION_INTERVAL bytes (default 1GB) > have been allocated in total, which may take a long time if the > leak is small relative to normal allocation churn. > - HEAP_PROFILE_ALLOCATION_INTERVAL > Leave at default (1GB) or lower for more frequent dumps early on. > Each dump is small (KB to low MB), so err toward more frequent. > - Optional: HEAP_PROFILE_INUSE_INTERVAL to also dump on new in-use > memory peaks. > > 3. Restart under HA > -------------------- > LD_PRELOAD requires a process restart -- cannot be applied with > `scontrol reconfigure`. > > If running primary/backup (SlurmctldHost[0]/SlurmctldHost[1]): > 1. Apply the drop-in to the BACKUP controller first, restart it, > confirm `sdiag` / `scontrol ping` look healthy and it is acting > as a normal backup. > 2. Manually fail over (`scontrol takeover`) or wait for a maintenance > window, then apply to primary and restart it. > 3. This gets profiling running on whichever instance is acting > primary with only the brief unavailability normal to any > slurmctld restart. > > If single-controller, this needs a scheduled maintenance window -- > same downtime as any ordinary slurmctld restart/upgrade, nothing > extra caused by the profiler itself. > > 4. Let it run > -------------- > Let it accumulate dumps for as long as it takes to see meaningful RSS > growth -- hours to a few days depending on leak rate. Files land as: > /var/spool/slurmctld/heapprof/slurmctld.hprof.0001.heap > /var/spool/slurmctld/heapprof/slurmctld.hprof.0002.heap > ... > Watch disk usage; each file is small, but keep an eye on the spool > directory. > > 5. Analyze > ----------- > Copy a few dumps spanning the growth window to an analysis host along > with the exact slurmctld binary you're are running (with symbols -- if > stripped, grab the matching -debuginfo/-dbgsym package for that build; > no rebuild needed). > > # Cumulative in-use memory at a single point, top offenders: > pprof --text /usr/sbin/slurmctld slurmctld.hprof.0010.heap > > # THE useful one -- diff an early snapshot against a late snapshot > # to isolate what grew between the two, filtering out steady-state > # background usage: > google-pprof --base=slurmctld.hprof.0002.heap \ > /usr/sbin/slurmctld slurmctld.hprof.0020.heap --text > > # Visual call graph of the diff (needs graphviz): > google-pprof --base=slurmctld.hprof.0002.heap \ > /usr/sbin/slurmctld slurmctld.hprof.0020.heap --pdf > growth.pdf > > The --base diff is the payoff -- it shows exactly which call stacks > accumulated allocations between the two snapshots, in bytes and object > count, sorted descending. That is the leak site, directly. > > 6. Revert when done > --------------------- > rm /etc/systemd/system/slurmctld.service.d/heapprofile.conf > systemctl daemon-reload > systemctl restart slurmctld # backup first, then failover back if desired > > Overhead / safety notes > ------------------------ > - Adds roughly 5-15% CPU/memory bookkeeping overhead at these settings > -- much lighter than valgrind (10-50x) and safe for a live scheduling > controller, but still worth doing on the backup first / during a > lower-traffic period if there is any concern. > - No changes to the Slurm build -- same RPM/binary, just LD_PRELOAD + > env vars via the systemd drop-in. > - If the controller is memory-constrained already, the profiler's own > bookkeeping adds some overhead on top of RSS -- usually not > significant, but worth noting if you are near a memory ceiling. > > =============================================================================== > > -Sheldon > ------------------------------------------------------------------------ > You are receiving this mail because: > > * You reported the ticket. >
I tried to run with Heap Profiling this morning, unfortunately the performance hit was too much to sustain. I'm not sure if it was the version of the tool in Rocky 8 (looks to be version 2.7) or just how busy our cluster is, but it was considerably more than 5-15% performance drag. The scheduler was pretty much pegged at 128 threads and was spending all its time answering communications instead of actually scheduling. User commands were super sluggish and sometimes failed due to time out, also munge started throwing time sync errors meaning that the messages coming in from the nodes were so backlogged that by the time that slurmctld got to them they were already outside the time window. Due to that performance drag I couldn't continue running in that mode so I reverted back to normal operating. Do you have a more light weight mode of monitoring? -Paul Edmon- On 8/20/2026 3:26 PM, bugs@schedmd.com wrote: > > *Comment # 13 <https://support.schedmd.com/show_bug.cgi?id=25685#c13> > on ticket 25685 <https://support.schedmd.com/show_bug.cgi?id=25685> > from Sheldon Teerlink <mailto:steerlink@nvidia.com> * > Hi Paul, > > Thanks for the update. 40GB RSS seems high to me. Can you please check your > MinJobAge value in slurm.conf and let me know what it is? A high value keeps > completed jobs in memory for a while. Also, can you determine how deep the job > queue is, as that will occupy memory? > > > squeue -h -t all | wc -l > > While it's good to know the memory plateaued, there could still be a leak. The > following are instructions on how to profile slurmctld and capture periodic > heap profiles using tcmalloc. If you're able to do this and provide the > analysis between heap dumps while memory is growing that would really help us > locate the memory growth. > > The slurmctld you're running has symbols so we'll get the information we need. > > > Heap Profiling slurmctld in Production with tcmalloc (gperftools) > ==================================================================== > > Works via LD_PRELOAD against the existing slurmctld binary -- no rebuild > or instrumented build required. > > The general idea here is to track all memory allocations made by > slurmctld and then analyze them over time. This requires restarting > the slurmctld process as described in this document. > > Adjust the paths and values as needed. > > 1. Install > ---------- > RHEL/Rocky: > dnf install gperftools gperftools-libs gperftools-devel > > Debian/Ubuntu: > apt install google-perftools libgoogle-perftools-dev > > Confirm the library is present: > find / -name 'libtcmalloc_and_profiler.so*' 2>/dev/null > > Use libtcmalloc_and_profiler.so (not plain libtcmalloc.so) -- it bundles > the heap profiler. > > 2. Wire it into the slurmctld systemd unit via a drop-in > ---------------------------------------------------------- > Do not edit the shipped unit file directly. Create an override instead > ensuring the correct paths are used: > > mkdir -p /etc/systemd/system/slurmctld.service.d > cat > /etc/systemd/system/slurmctld.service.d/heapprofile.conf <<'EOF' > [Service] > > Environment=LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libtcmalloc_and_profiler.so.4 > Environment=HEAPPROFILE=/var/spool/slurmctld/heapprof/slurmctld.hprof > Environment=HEAP_PROFILE_TIME_INTERVAL=3600 > Environment=HEAP_PROFILE_ALLOCATION_INTERVAL=1073741824 > EOF > mkdir -p /var/spool/slurmctld/heapprof > systemctl daemon-reload > > Key env vars: > - HEAP_PROFILE_TIME_INTERVAL=3600 > Forces a dump every hour regardless of allocation volume. This is > the one that matters for a slow leak -- without it, dumps only > fire after HEAP_PROFILE_ALLOCATION_INTERVAL bytes (default 1GB) > have been allocated in total, which may take a long time if the > leak is small relative to normal allocation churn. > - HEAP_PROFILE_ALLOCATION_INTERVAL > Leave at default (1GB) or lower for more frequent dumps early on. > Each dump is small (KB to low MB), so err toward more frequent. > - Optional: HEAP_PROFILE_INUSE_INTERVAL to also dump on new in-use > memory peaks. > > 3. Restart under HA > -------------------- > LD_PRELOAD requires a process restart -- cannot be applied with > `scontrol reconfigure`. > > If running primary/backup (SlurmctldHost[0]/SlurmctldHost[1]): > 1. Apply the drop-in to the BACKUP controller first, restart it, > confirm `sdiag` / `scontrol ping` look healthy and it is acting > as a normal backup. > 2. Manually fail over (`scontrol takeover`) or wait for a maintenance > window, then apply to primary and restart it. > 3. This gets profiling running on whichever instance is acting > primary with only the brief unavailability normal to any > slurmctld restart. > > If single-controller, this needs a scheduled maintenance window -- > same downtime as any ordinary slurmctld restart/upgrade, nothing > extra caused by the profiler itself. > > 4. Let it run > -------------- > Let it accumulate dumps for as long as it takes to see meaningful RSS > growth -- hours to a few days depending on leak rate. Files land as: > /var/spool/slurmctld/heapprof/slurmctld.hprof.0001.heap > /var/spool/slurmctld/heapprof/slurmctld.hprof.0002.heap > ... > Watch disk usage; each file is small, but keep an eye on the spool > directory. > > 5. Analyze > ----------- > Copy a few dumps spanning the growth window to an analysis host along > with the exact slurmctld binary you're are running (with symbols -- if > stripped, grab the matching -debuginfo/-dbgsym package for that build; > no rebuild needed). > > # Cumulative in-use memory at a single point, top offenders: > pprof --text /usr/sbin/slurmctld slurmctld.hprof.0010.heap > > # THE useful one -- diff an early snapshot against a late snapshot > # to isolate what grew between the two, filtering out steady-state > # background usage: > google-pprof --base=slurmctld.hprof.0002.heap \ > /usr/sbin/slurmctld slurmctld.hprof.0020.heap --text > > # Visual call graph of the diff (needs graphviz): > google-pprof --base=slurmctld.hprof.0002.heap \ > /usr/sbin/slurmctld slurmctld.hprof.0020.heap --pdf > growth.pdf > > The --base diff is the payoff -- it shows exactly which call stacks > accumulated allocations between the two snapshots, in bytes and object > count, sorted descending. That is the leak site, directly. > > 6. Revert when done > --------------------- > rm /etc/systemd/system/slurmctld.service.d/heapprofile.conf > systemctl daemon-reload > systemctl restart slurmctld # backup first, then failover back if desired > > Overhead / safety notes > ------------------------ > - Adds roughly 5-15% CPU/memory bookkeeping overhead at these settings > -- much lighter than valgrind (10-50x) and safe for a live scheduling > controller, but still worth doing on the backup first / during a > lower-traffic period if there is any concern. > - No changes to the Slurm build -- same RPM/binary, just LD_PRELOAD + > env vars via the systemd drop-in. > - If the controller is memory-constrained already, the profiler's own > bookkeeping adds some overhead on top of RSS -- usually not > significant, but worth noting if you are near a memory ceiling. > > =============================================================================== > > -Sheldon > ------------------------------------------------------------------------ > You are receiving this mail because: > > * You reported the ticket. >
Hi Paul, There are a few other things to try with tcmalloc before trying another tool (jemalloc). The setup I sent yesterday combines allocation with profiling. We need to know which one is the problem. Also tcmalloc 2.7 is a bit old (2018 maybe). Here are some instructions to separate the memory allocation from the profiling. If we can get allocation to work properly then we can add back in profiling/sampling at a lower rate. I'll attached the tuning instructions. -Sheldon
Created attachment 46081 [details] Instructions for profiling memory usage
Thanks. I will give this a try. Do you think it worth building a newer version of gperftools? I was glancing through their change log but I didn't see anything yet that indicates a massive performance improvement. -Paul Edmon- On 8/21/2026 11:24 AM, bugs@schedmd.com wrote: > > *Comment # 17 <https://support.schedmd.com/show_bug.cgi?id=25685#c17> > on ticket 25685 <https://support.schedmd.com/show_bug.cgi?id=25685> > from Sheldon Teerlink <mailto:steerlink@nvidia.com> * > Createdattachment 46081 <attachment.cgi?id=46081> [details] > <attachment.cgi?id=46081&action=edit> > Instructions for profiling memory usage > ------------------------------------------------------------------------ > You are receiving this mail because: > > * You reported the ticket. >
(In reply to Paul Edmon from comment #18) > Thanks. I will give this a try. > > Do you think it worth building a newer version of gperftools? I was > glancing through their change log but I didn't see anything yet that > indicates a massive performance improvement. > > -Paul Edmon- > > On 8/21/2026 11:24 AM, bugs@schedmd.com wrote: > > > > *Comment # 17 <https://support.schedmd.com/show_bug.cgi?id=25685#c17> > > on ticket 25685 <https://support.schedmd.com/show_bug.cgi?id=25685> > > from Sheldon Teerlink <mailto:steerlink@nvidia.com> * > > Createdattachment 46081 <attachment.cgi?id=46081> [details] > > <attachment.cgi?id=46081&action=edit> > > Instructions for profiling memory usage > > ------------------------------------------------------------------------ > > You are receiving this mail because: > > > > * You reported the ticket. > > I don't think it is worth building a newer version at this point. Let's see if tuning tcmalloc works and then we can try another tool.
Okay, I tried out your tuning instructions. I can confirm it is the profiler that is the slow part as with minimal it ran just fine. Unfortunately I tried dialing up the Sample Parameter to 32MB no dice, its still as slow as molasses and the scheduler is pegged out at 128 threads which means backfill scheduling has ground to a halt. We will either need to dial back the profiling more or swap tools. What do you suggest? -Paul Edmon- On 8/21/2026 11:32 AM, bugs@schedmd.com wrote: > > *Comment # 19 <https://support.schedmd.com/show_bug.cgi?id=25685#c19> > on ticket 25685 <https://support.schedmd.com/show_bug.cgi?id=25685> > from Sheldon Teerlink <mailto:steerlink@nvidia.com> * > (In reply to Paul Edmon fromcomment #18 <show_bug.cgi?id=25685#c18>) > > Thanks. I will give this a try. > > Do you think it worth building a newer version of gperftools? I > was > glancing through their change log but I didn't see anything yet > that > indicates a massive performance improvement. > > -Paul Edmon- > > > On 8/21/2026 11:24 AM, bugs@schedmd.com wrote: > > > > *Comment # 17 > <show_bug.cgi?id=25685#c17> > <https://support.schedmd.com/show_bug.cgi?id=25685#c17 > <show_bug.cgi?id=25685#c17>> > > on ticket 25685 > <show_bug.cgi?id=25685> > <https://support.schedmd.com/show_bug.cgi?id=25685 > <show_bug.cgi?id=25685>> > > from Sheldon Teerlink > <mailto:steerlink@nvidia.com <mailto:steerlink@nvidia.com>> * > > > Createdattachment 46081 <attachment.cgi?id=46081> [details] > > > <attachment.cgi?id=46081&action=edit> > > Instructions for profiling > memory usage > > > ------------------------------------------------------------------------ > > > You are receiving this mail because: > > > > * You reported the > ticket. > > > > I don't think it is worth building a newer version at this point. Let's see if > tuning tcmalloc works and then we can try another tool. > ------------------------------------------------------------------------ > You are receiving this mail because: > > * You reported the ticket. >
If you've tried both > Environment=TCMALLOC_SAMPLE_PARAMETER=33554432 > Environment=TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=536870912 with no success then I recommend we try jemalloc. I can prepare some instructions and send them over.
Yeah, lets try swapping to jemalloc. I won't have time to run it though until Monday as we are entering our weekend change freeze. -Paul Edmon- On 8/21/2026 12:09 PM, bugs@schedmd.com wrote: > > *Comment # 21 <https://support.schedmd.com/show_bug.cgi?id=25685#c21> > on ticket 25685 <https://support.schedmd.com/show_bug.cgi?id=25685> > from Sheldon Teerlink <mailto:steerlink@nvidia.com> * > If you've tried both > > > Environment=TCMALLOC_SAMPLE_PARAMETER=33554432 > Environment=TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=536870912 > > with no success then I recommend we try jemalloc. > > I can prepare some instructions and send them over. > ------------------------------------------------------------------------ > You are receiving this mail because: > > * You reported the ticket. >
Sounds good. I'll have those instructions over to you before Monday. (In reply to Paul Edmon from comment #22) > Yeah, lets try swapping to jemalloc. I won't have time to run it though > until Monday as we are entering our weekend change freeze. > > -Paul Edmon- > > On 8/21/2026 12:09 PM, bugs@schedmd.com wrote: > > > > *Comment # 21 <https://support.schedmd.com/show_bug.cgi?id=25685#c21> > > on ticket 25685 <https://support.schedmd.com/show_bug.cgi?id=25685> > > from Sheldon Teerlink <mailto:steerlink@nvidia.com> * > > If you've tried both > > > > > Environment=TCMALLOC_SAMPLE_PARAMETER=33554432 > Environment=TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=536870912 > > > > with no success then I recommend we try jemalloc. > > > > I can prepare some instructions and send them over. > > ------------------------------------------------------------------------ > > You are receiving this mail because: > > > > * You reported the ticket. > >
Created attachment 46082 [details] Memory Profiling using jemalloc
Hi Paul, I sent over the instructions for profiling with jemalloc. The tuning knobs are different than jemalloc. -Sheldon
Thanks. I will try this out as I have time next week. -Paul Edmon- On 8/21/2026 7:00 PM, bugs@schedmd.com wrote: > > *Comment # 25 <https://support.schedmd.com/show_bug.cgi?id=25685#c25> > on ticket 25685 <https://support.schedmd.com/show_bug.cgi?id=25685> > from Sheldon Teerlink <mailto:steerlink@nvidia.com> * > Hi Paul, > > I sent over the instructions for profiling with jemalloc. The tuning knobs are > different than jemalloc. > > -Sheldon > ------------------------------------------------------------------------ > You are receiving this mail because: > > * You reported the ticket. >
Just wanted to log this here. The growth in slurmctld got up to 100GB this weekend before I had to restart slurmctld. So clearly we still have a slow memory leak somewhere instead of slurmctld eating more space than it used to. I will work on instrumenting it on Monday when I have time. -Paul Edmon- On 8/21/2026 7:00 PM, bugs@schedmd.com wrote: > > *Comment # 25 <https://support.schedmd.com/show_bug.cgi?id=25685#c25> > on ticket 25685 <https://support.schedmd.com/show_bug.cgi?id=25685> > from Sheldon Teerlink <mailto:steerlink@nvidia.com> * > Hi Paul, > > I sent over the instructions for profiling with jemalloc. The tuning knobs are > different than jemalloc. > > -Sheldon > ------------------------------------------------------------------------ > You are receiving this mail because: > > * You reported the ticket. >
Looks like it is working this time with jemalloc. Keep you posted what I find. -Paul Edmon- On 8/21/26 7:00 PM, bugs@schedmd.com wrote: > > *Comment # 25 <https://support.schedmd.com/show_bug.cgi?id=25685#c25> > on ticket 25685 <https://support.schedmd.com/show_bug.cgi?id=25685> > from Sheldon Teerlink <mailto:steerlink@nvidia.com> * > Hi Paul, > > I sent over the instructions for profiling with jemalloc. The tuning knobs are > different than jemalloc. > > -Sheldon > ------------------------------------------------------------------------ > You are receiving this mail because: > > * You reported the ticket. >
Successfully ran jemalloc yesterday and I have the results (ended up producing 2.5 million files and 120 GB of data). Digging into to it I selected the following two dumps: 2026-08-24 12:48:41 31191912 kB 148 31276 [root@holy-slurm02 heapprof]# /opt/jemalloc-prof/bin/jeprof --show_bytes --text /usr/sbin/slurmctld jeprof.1160171.895331.i895331.heap Using local file /usr/sbin/slurmctld. Using local file jeprof.1160171.895331.i895331.heap. Total: 30034633023 B 29907707491 99.6% 99.6% 29907707491 99.6% slurm_xcalloc 67131443 0.2% 99.8% 67131443 0.2% slurm_xrecalloc 43014807 0.1% 99.9% 43014807 0.1% luaopen_math 8390656 0.0% 100.0% 8390656 0.0% s2n_map_iterator_next 8388624 0.0% 100.0% 8388624 0.0% CRYPTO_zalloc 2026-08-24 21:26:39 58294120 kB 154 28867 [root@holy-slurm02 heapprof]# /opt/jemalloc-prof/bin/jeprof --show_bytes --text /usr/sbin/slurmctld jeprof.1160171.2424158.i2424158.heap Using local file /usr/sbin/slurmctld. Using local file jeprof.1160171.2424158.i2424158.heap. Total: 56714988811 B 56647750986 99.9% 99.9% 56647750986 99.9% slurm_xcalloc 50458545 0.1% 100.0% 50458545 0.1% slurm_xrecalloc 8390656 0.0% 100.0% 8390656 0.0% s2n_map_iterator_next 8388624 0.0% 100.0% 8388624 0.0% CRYPTO_zalloc Which shows slurm_xcalloc having the bulk of the usage. When I produced the diff pdf (which I will attach) it narrowed it down to list_create. From there I grepped the diff and got: [root@holy-slurm02 heapprof]# /opt/jemalloc-prof/bin/jeprof --show_bytes --base=jeprof.1160171.895331.i895331.heap --lines --text --focus="list_create" /usr/sbin/slurmctld jeprof.1160171.2424158.i2424158.heap Using local file /usr/sbin/slurmctld. Using local file jeprof.1160171.2424158.i2424158.heap. Total: 26680355788 B 25465360322 97.7% 97.7% 25465360322 97.7% slurm_xcalloc /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/common/xmalloc.c:97 595592872 2.3% 100.0% 595592872 2.3% slurm_xcalloc /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/common/xmalloc.c:99 0 0.0% 100.0% 26086125163 100.1% __GI___clone :? 0 0.0% 100.0% -25171968 -0.1% __libc_start_main ??:? 0 0.0% 100.0% 26102906475 100.2% _attempt_backfill /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/plugins/sched/backfill/backfill.c:3274 0 0.0% 100.0% -8390656 -0.0% _attempt_backfill /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/plugins/sched/backfill/backfill.c:3414 0 0.0% 100.0% 704647776 2.7% _build_sock_gres_by_topo (inline) /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/plugins/select/cons_tres/gres_sock_list.c:245 0 0.0% 100.0% 268435968 1.0% _build_sock_gres_by_topo (inline) /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/plugins/select/cons_tres/gres_sock_list.c:247 0 0.0% 100.0% 201326976 0.8% _build_sock_gres_by_topo (inline) /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/plugins/select/cons_tres/gres_sock_list.c:248 0 0.0% 100.0% 595592872 2.3% _build_sock_gres_by_topo (inline) /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/plugins/select/cons_tres/gres_sock_list.c:416 0 0.0% 100.0% 26102906475 100.2% _can_job_run_on_node /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/plugins/select/cons_tres/job_test.c:566 0 0.0% 100.0% 50343937 0.2% _copy_job_desc_to_job_record (inline) /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/slurmctld/job_mgr.c:8707 0 0.0% 100.0% 25171968 0.1% _create_job_record (inline) /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/slurmctld/job_mgr.c:821 0 0.0% 100.0% 1770003592 6.8% _foreach_gres_sock_list_create /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/plugins/select/cons_tres/gres_sock_list.c:894 0 0.0% 100.0% -8390656 -0.0% _foreach_job_state_extract /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/interfaces/gres.c:7655 0 0.0% 100.0% 8390656 0.0% _foreach_select_nodes_part_list /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/slurmctld/job_mgr.c:4044 (discriminator 3) 0 0.0% 100.0% 7903877395 30.3% _future_run_test (inline) /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/plugins/select/cons_tres/job_test.c:2773 0 0.0% 100.0% 18190638424 69.8% _future_run_test (inline) /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/plugins/select/cons_tres/job_test.c:2877 0 0.0% 100.0% 25171968 0.1% _get_job_parts /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/slurmctld/job_mgr.c:6647 0 0.0% 100.0% 26102906475 100.2% _get_one_res (inline) /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/plugins/select/cons_tres/job_test.c:846 0 0.0% 100.0% 25171968 0.1% _job_create /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/slurmctld/job_mgr.c:7372 0 0.0% 100.0% 8390656 0.0% _job_create /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/slurmctld/job_mgr.c:7487 0 0.0% 100.0% 50343937 0.2% _job_create /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/slurmctld/job_mgr.c:7607 0 0.0% 100.0% 26102906475 100.2% _job_test /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/plugins/select/cons_tres/job_test.c:1578 0 0.0% 100.0% 8390656 0.0% _list_find_first_lock /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/common/list.c:473 (discriminator 6) 0 0.0% 100.0% 8390656 0.0% _list_find_first_locked /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/common/list.c:454 0 0.0% 100.0% 92297217 0.4% _on_extract /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/slurmctld/controller.c:2081 0 0.0% 100.0% 26086125163 100.1% _run /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/common/threadpool.c:437 0 0.0% 100.0% -8390656 -0.0% _run_now (inline) /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/plugins/select/cons_tres/job_test.c:3073 0 0.0% 100.0% 92297217 0.4% _run_work /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/conmgr/work.c:288 0 0.0% 100.0% -100687874 -0.4% _sched_agent /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/slurmctld/job_scheduler.c:959 0 0.0% 100.0% -100687874 -0.4% _schedule (inline) /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/slurmctld/job_scheduler.c:1798 0 0.0% 100.0% 26102906475 100.2% _select_nodes /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/plugins/select/cons_tres/job_test.c:1026 0 0.0% 100.0% 8390656 0.0% _select_nodes_base /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/slurmctld/job_mgr.c:3902 0 0.0% 100.0% 8390656 0.0% _select_nodes_parts (inline) /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/slurmctld/job_mgr.c:4073 0 0.0% 100.0% 8390656 0.0% _select_nodes_qos /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/slurmctld/job_mgr.c:4025 0 0.0% 100.0% 8390656 0.0% _select_nodes_resvs /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/slurmctld/job_mgr.c:3990 0 0.0% 100.0% 92297217 0.4% _service_connection /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/slurmctld/controller.c:2018 0 0.0% 100.0% 92297217 0.4% _slurm_rpc_submit_batch_job /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/slurmctld/proc_req.c:3940 0 0.0% 100.0% -25171968 -0.1% _start ??:? 0 0.0% 100.0% -8390656 -0.0% _start_job /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/plugins/sched/backfill/backfill.c:3911 0 0.0% 100.0% -8390656 -0.0% _sync_jobs_to_conf /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/slurmctld/read_config.c:1368 0 0.0% 100.0% 8390656 0.0% _test_only (inline) /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/plugins/select/cons_tres/job_test.c:2365 0 0.0% 100.0% 26086125163 100.1% _thread /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/common/threadpool.c:583 0 0.0% 100.0% 26094515819 100.1% _try_sched /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/plugins/sched/backfill/backfill.c:636 0 0.0% 100.0% 8390656 0.0% _try_sched /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/plugins/sched/backfill/backfill.c:662 (discriminator 4) 0 0.0% 100.0% 8390656 0.0% _will_run_test (inline) /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/plugins/select/cons_tres/job_test.c:2978 0 0.0% 100.0% 26094515819 100.1% _will_run_test (inline) /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/plugins/select/cons_tres/job_test.c:3006 0 0.0% 100.0% 92297217 0.4% _worker /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/conmgr/workers.c:297 (discriminator 2) 0 0.0% 100.0% 26094515819 100.1% backfill_agent /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/plugins/sched/backfill/backfill.c:1201 0 0.0% 100.0% 595592872 2.3% bit_alloc_nz /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/common/bitstring.c:326 0 0.0% 100.0% -16781312 -0.1% depended_list_copy /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/slurmctld/job_scheduler.c:3095 0 0.0% 100.0% -8390656 -0.0% feature_list_copy /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/slurmctld/job_scheduler.c:5126 0 0.0% 100.0% 16781312 0.1% get_part_list /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/slurmctld/partition_mgr.c:710 0 0.0% 100.0% -8390656 -0.0% gres_job_state_extract /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/interfaces/gres.c:7678 0 0.0% 100.0% -8390656 -0.0% gres_job_state_list_dup /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/interfaces/gres.c:7443 0 0.0% 100.0% 8390656 0.0% gres_job_state_validate /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/interfaces/gres.c:7002 0 0.0% 100.0% 24332902883 93.4% gres_sock_list_create /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/plugins/select/cons_tres/gres_sock_list.c:952 0 0.0% 100.0% 1770003592 6.8% gres_sock_list_create /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/plugins/select/cons_tres/gres_sock_list.c:953 0 0.0% 100.0% 16781312 0.1% gres_stepmgr_job_alloc /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/stepmgr/gres_stepmgr.c:1087 0 0.0% 100.0% 83906561 0.3% job_allocate /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/slurmctld/job_mgr.c:4221 0 0.0% 100.0% 8390656 0.0% job_allocate /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/slurmctld/job_mgr.c:4376 0 0.0% 100.0% -117469186 -0.5% job_array_post_sched /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/slurmctld/job_mgr.c:19362 0 0.0% 100.0% -25171968 -0.1% job_array_split /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/slurmctld/job_mgr.c:3480 0 0.0% 100.0% -58734593 -0.2% job_array_split /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/slurmctld/job_mgr.c:3615 0 0.0% 100.0% -16781312 -0.1% job_array_split /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/slurmctld/job_mgr.c:3708 0 0.0% 100.0% -8390656 -0.0% job_array_split /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/slurmctld/job_mgr.c:3725 0 0.0% 100.0% -8390656 -0.0% job_array_split /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/slurmctld/job_mgr.c:3796 0 0.0% 100.0% -16781312 -0.1% job_mgr_load_job_state /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/slurmctld/job_mgr.c:1510 0 0.0% 100.0% 8390656 0.0% job_record_create /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/common/job_record.c:69 0 0.0% 100.0% -16781312 -0.1% job_record_unpack /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/common/job_record.c:3609 0 0.0% 100.0% 16781312 0.1% job_res_add_job /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/plugins/select/cons_tres/job_resources.c:313 0 0.0% 100.0% 26102906475 100.2% job_test /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/plugins/select/cons_tres/job_test.c:4051 0 0.0% 100.0% 8390656 0.0% job_test /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/plugins/select/cons_tres/job_test.c:4059 0 0.0% 100.0% -8390656 -0.0% job_test /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/plugins/select/cons_tres/job_test.c:4062 0 0.0% 100.0% 24290949602 93.2% list_create /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/common/list.c:196 0 0.0% 100.0% 8390656 0.0% list_find_first /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/common/list.c:482 0 0.0% 100.0% 1761612935 6.8% list_for_each /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/common/list.c:614 0 0.0% 100.0% 1761612935 6.8% list_for_each_max /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/common/list.c:645 0 0.0% 100.0% -16781312 -0.1% load_all_job_state /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/slurmctld/job_mgr.c:1398 0 0.0% 100.0% -25171968 -0.1% main /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/slurmctld/controller.c:1028 0 0.0% 100.0% 92297217 0.4% on_extract /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/conmgr/con.c:2074 0 0.0% 100.0% -58734593 -0.2% part_list_copy /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/slurmctld/partition_mgr.c:664 0 0.0% 100.0% -16781312 -0.1% read_slurm_conf /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/slurmctld/read_config.c:1730 0 0.0% 100.0% -8390656 -0.0% read_slurm_conf /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/slurmctld/read_config.c:1782 0 0.0% 100.0% 26102906475 100.2% select_g_job_test /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/interfaces/select.c:284 0 0.0% 100.0% 16781312 0.1% select_g_select_nodeinfo_set /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/interfaces/select.c:378 0 0.0% 100.0% -117469186 -0.5% select_nodes /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/slurmctld/node_scheduler.c:2939 0 0.0% 100.0% 16781312 0.1% select_nodes /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/slurmctld/node_scheduler.c:3004 0 0.0% 100.0% 26102906475 100.2% select_p_job_test /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/plugins/select/cons_tres/select_cons_tres.c:351 0 0.0% 100.0% 16781312 0.1% select_p_select_nodeinfo_set /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/plugins/select/cons_tres/select_cons_tres.c:924 0 0.0% 100.0% 92297217 0.4% slurmctld_req /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/slurmctld/proc_req.c:7462 0 0.0% 100.0% 26086125163 100.1% start_thread ??:? [root@holy-slurm02 heapprof]# /opt/jemalloc-prof/bin/jeprof --show_bytes --base=jeprof.1160171.895331.i895331.heap --lines --text /usr/sbin/slurmctld jeprof.1160171.2424158.i2424158.heap | grep list_create Using local file /usr/sbin/slurmctld. Using local file jeprof.1160171.2424158.i2424158.heap. 0 0.0% 100.2% 1770003592 6.6% _foreach_gres_sock_list_create /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/plugins/select/cons_tres/gres_sock_list.c:894 0 0.0% 100.2% 24332902883 91.2% gres_sock_list_create /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/plugins/select/cons_tres/gres_sock_list.c:952 0 0.0% 100.2% 1770003592 6.6% gres_sock_list_create /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/plugins/select/cons_tres/gres_sock_list.c:953 0 0.0% 100.2% 24290949602 91.0% list_create /home/gitlab-runner/builds/DY5eBZ31r/0/fasrc/ops/yum-repos/rpmbuild/BUILD/slurm-26.05.3/src/common/list.c:196 So that looks to be our culprit. What specifically is causing the leak I cannot say but that's where the bulk of the memory is being used up. Let me know what other information you need to narrow this down as I am happy to provide it.
Created attachment 46086 [details] Memory graph for jemalloc run in 2026-08-24
Created attachment 46087 [details] Growth chart for jemalloc [root@holy-slurm02 heapprof]# /opt/jemalloc-prof/bin/jeprof --show_bytes --base=jeprof.1160171.895331.i895331.heap --pdf /usr/sbin/slurmctld jeprof.1160171.2424158.i2424158.heap > growth.pdf Using local file /usr/sbin/slurmctld. Using local file jeprof.1160171.2424158.i2424158.heap. Dropping nodes with <= 133401778 B; edges with <= 26680355 abs(B)
By the way for reference the numbers I quoted earlier are: Time RSS Threads Job Count 2026-08-24 12:48:41 31191912 kB 148 31276 2026-08-24 21:26:39 58294120 kB 154 28867 So you can see that the memory growth is not correlated with thread count or job count.
Hi Paul, This heap profile data is extremely useful. Thank you for collecting it. I belive I've identified the exact cause of the memory leak and will start working on a fix. The leak happens when jobs request a "mode_3" HRES license. Can you confirm you're using HRES? Also, do you see messages like these in the logs? > "Test fail on node %s: hres_select_find_leaf" -Sheldon
We actually are not using hres to my knowledge. We do use topology.conf, constraints, TRES, but we don't have a resources.yaml set. I don't see any errors either related to hres. What I do see are stuff like: Aug 25 13:39:03 holy-slurm02 slurmctld[3756932]: error: auth_p_get_host: Lookup failed for 0.0.0.0 Aug 25 13:39:05 holy-slurm02 slurmctld[3756932]: error: auth_p_get_host: Lookup failed for 0.0.0.0 Aug 25 13:39:06 holy-slurm02 slurmctld[3756932]: error: auth_p_get_host: Lookup failed for 0.0.0.0 Aug 25 13:39:08 holy-slurm02 slurmctld[3756932]: error: auth_p_get_host: Lookup failed for 0.0.0.0 Aug 25 13:39:10 holy-slurm02 slurmctld[3756932]: error: _handle_qos_tres_run_secs: job 41796064: QOS normal TRES cpu account used limit tres_run_secs underflow, tried to remove 3183 seconds when only 3123 remained. Aug 25 13:39:10 holy-slurm02 slurmctld[3756932]: error: _handle_qos_tres_run_secs: job 41796064: QOS normal TRES cpu user used limit tres_run_secs underflow, tried to remove 3183 seconds when only 3123 remained. Aug 25 13:39:10 holy-slurm02 slurmctld[3756932]: error: _handle_qos_tres_run_secs: job 41796064: QOS normal TRES mem account used limit tres_run_secs underflow, tried to remove 954900 seconds when only 936900 remained. Aug 25 13:39:10 holy-slurm02 slurmctld[3756932]: error: _handle_qos_tres_run_secs: job 41796064: QOS normal TRES mem user used limit tres_run_secs underflow, tried to remove 954900 seconds when only 936900 remained. Aug 25 13:39:10 holy-slurm02 slurmctld[3756932]: error: _handle_qos_tres_run_secs: job 41796064: QOS normal TRES node account used limit tres_run_secs underflow, tried to remove 3183 seconds when only 3123 remained. Aug 25 13:39:10 holy-slurm02 slurmctld[3756932]: error: _handle_qos_tres_run_secs: job 41796064: QOS normal TRES node user used limit tres_run_secs underflow, tried to remove 3183 seconds when only 3123 remained. Aug 25 13:39:10 holy-slurm02 slurmctld[3756932]: error: _handle_assoc_tres_run_secs: job 41796064: assoc 15247 TRES cpu grp_used_tres_run_secs underflow, tried to remove 3183 seconds when only 3123 remained. Aug 25 13:40:32 holy-slurm02 slurmctld[3756932]: error: slurm_send_node_msg: [socket:[1836337492]] slurm_bufs_sendto(msg_type=RESPONSE_SLURM_RC) failed: Unexpected missing socket error But those have been there for a while. Anyways perhaps our conf is intersecting the HRES stuff but we don't use it explicitly. -Paul Edmon- On 8/25/26 1:38 PM, bugs@schedmd.com wrote: > > *Comment # 33 <https://support.schedmd.com/show_bug.cgi?id=25685#c33> > on ticket 25685 <https://support.schedmd.com/show_bug.cgi?id=25685> > from Sheldon Teerlink <mailto:steerlink@nvidia.com> * > Hi Paul, > > This heap profile data is extremely useful. Thank you for collecting it. I > belive I've identified the exact cause of the memory leak and will start > working on a fix. > > The leak happens when jobs request a "mode_3" HRES license. Can you confirm > you're using HRES? > > Also, do you see messages like these in the logs? > > > "Test fail on node %s: hres_select_find_leaf" > > -Sheldon > ------------------------------------------------------------------------ > You are receiving this mail because: > > * You reported the ticket. >
Created attachment 46088 [details] Current slurm.conf
Created attachment 46089 [details] Current topology.conf
For reference I've dropped our current slurm.conf and topology.conf. As noted we have no resources.yaml set so if we are using HRES it is inadvertant.
Thanks for checking on the HRES. I've searched higher in the call tree on the hot path and have found another code path, this one involving topolgy, that leaks memory. It is also a code path that was introduced in 26.05 so that seems much more likely.
Hi Paul, I've submitted a fix and it is under review. I'm not sure exactly when it will be officially available, but should be soon. -Sheldon
Let me know if you want a source code patch for version 26.05.3 that you can test locally. -Sheldon
Right now I have things under control by restarting the scheduler every 6 hours. If it won't come out in 26.05.4 I would prefer to get it sooner if only to stop needing to restart the scheduler every 6 hours (which could lead to other issues). That said if you need to verify that the patch actually solved the problem I am game to apply it here. I don't know if you managed to replicate on your end and test. If so then probably no need to test on my end as it sounds like you figured out what it was. -Paul Edmon- On 8/25/2026 6:52 PM, bugs@schedmd.com wrote: > > *Comment # 41 <https://support.schedmd.com/show_bug.cgi?id=25685#c41> > on ticket 25685 <https://support.schedmd.com/show_bug.cgi?id=25685> > from Sheldon Teerlink <mailto:steerlink@nvidia.com> * > Let me know if you want a source code patch for version 26.05.3 that you can > test locally. > > -Sheldon > ------------------------------------------------------------------------ > You are receiving this mail because: > > * You reported the ticket. >
Sounds good. I'll attach the patch and let you decide if you want to evaluate it. Will keep you posted as the fix moves through review. -Sheldon (In reply to Paul Edmon from comment #42) > Right now I have things under control by restarting the scheduler every > 6 hours. If it won't come out in 26.05.4 I would prefer to get it sooner > if only to stop needing to restart the scheduler every 6 hours (which > could lead to other issues). > > That said if you need to verify that the patch actually solved the > problem I am game to apply it here. I don't know if you managed to > replicate on your end and test. If so then probably no need to test on > my end as it sounds like you figured out what it was. > > -Paul Edmon- > > On 8/25/2026 6:52 PM, bugs@schedmd.com wrote: > > > > *Comment # 41 <https://support.schedmd.com/show_bug.cgi?id=25685#c41> > > on ticket 25685 <https://support.schedmd.com/show_bug.cgi?id=25685> > > from Sheldon Teerlink <mailto:steerlink@nvidia.com> * > > Let me know if you want a source code patch for version 26.05.3 that you can > > test locally. > > > > -Sheldon > > ------------------------------------------------------------------------ > > You are receiving this mail because: > > > > * You reported the ticket. > >
Created attachment 46104 [details] 26.05.3 memory leak fix patch
Hi Paul, The memory leak fix has passed review and is now in the slurm-26.05 and master branches. The 26.05.4 release should be available pretty soon. Let's keep this ticket open until you're able to install it and confirm the leak has been fixed. -Sheldon
Great. We were planning on upgrading to 26.05.4 on September 14th. So we should pick it up then. -Paul Edmon- On 8/31/26 11:48 AM, bugs@schedmd.com wrote: > > *Comment # 45 <https://support.schedmd.com/show_bug.cgi?id=25685#c45> > on ticket 25685 <https://support.schedmd.com/show_bug.cgi?id=25685> > from Sheldon Teerlink <mailto:steerlink@nvidia.com> * > Hi Paul, > > The memory leak fix has passed review and is now in the slurm-26.05 and master > branches. The 26.05.4 release should be available pretty soon. > > Let's keep this ticket open until you're able to install it and confirm the > leak has been fixed. > > -Sheldon > ------------------------------------------------------------------------ > You are receiving this mail because: > > * You reported the ticket. >
