Created attachment 30792 [details] Memory usage of slurmctld Hi, I am experiencing a problem with slurmrestd memory usage when job status is queried. On production system it grows beyond 12GB per query and it does not shrink even when REST query is done and client disconnects. OOM killer is always triggered eventually. See attached png as an example, it shows memory size of slurmctld and slurmrestd. How to replicate: - submit lots of jobs, the more the quicker slurmrestd grows in size, and make sur they stay queued for x in `seq 1 20000`; do echo -e '#!/bin/bash\necho A'|sbatch -H; done - run constant jobs query in a loop, example from Powershell While (1) { Invoke-RestMethod -Method Get -Headers $headers -Body $body -Uri 'http://10.10.10.10:6820/slurm/v0.0.38/jobs' } - run top and watch slurmrestd grow in size until it grabs all memory available Please note that it keeps growing in size even with one job but the more jobs in the queue the faster it grows. Is such behaviour expected? I am happy to provide more data when requested. Regards, Marcin
Hi, We are experiencing the same issue. We are using slurm 24.05.3 currently. Did you find a way to solve this or some workaround? --Tal
Closing this old ticket as it was likely fixed by ticket#17246 or ticket#16389. If this is still an issue, please open a new ticket on a currently supported version of Slurm.
The issue still persists. SLURM version: 24.11.5 REST API version: v0.0.41 Approximately 2,800 jobs were created using the same method as in the original case. On each /slurm/v0.0.41/jobs/ request, slurmrestd allocates around 500 MiB of memory — roughly 200 KB per job — and this memory is not released.
Maybe it will be helpful: there is no such leak for /nodes method, nor for /diag. Although we don't have such amount of nodes.
We experienced the same issue, and managed to locate the/an issue. It sounds similar to what you are experiencing so I hope what we found would help you: 1. Slurmrestd runs by default with 20 threads. 2. Each thread manages its own memory 3. Once a thread allocated space, it does not free it but only reuses it. This means, that if your query returns 2G of data, then each thread that ever handles that request (round robinned) will reach the 2G and never go down until the service is restarted. Hence it will grow to 40G total memory usage! However, a turnover of jobs will reuse the same 2G per thread and will not make it grow further. Our solution to this was to create 2 instances of slurmrestd - one default and one with '-t3' flag - which sets up only 3 threads. Then we set up an nginx instance that forewards requests to /jobs to the -t3 instance, and the rest of the requests to the default process. What we see is that the -t3 instance grows to around 6G (2g per thread) of memory, but stays there, and the default instance, as it gets only static sized requests such as nodes and diag, does not grow so much. Do notice: the more jobs you have in queue, the more space it takes *per thread* so we have 12G memory reserved for the -t3 instance. Which should allow us to up to 2 jobs in the queue then our standard. Also we monitor it, so if it grows close we can allocate more memory. Also, notice that slurmdb/*/jobs - will also return huge amounts of data, so make sure to manage how the access to it is done and send it to the -t3 instance. Hope this helps you!
Thanks, Tal — this is really helpful! Reducing the thread count made a noticeable difference. However, memory consumption still appears to scale linearly with the number of jobs known to the controller. In my case, 2,800 jobs resulted in a 20MB response from the /jobs handler and around 375MB of additional memory usage. That’s nearly an order of magnitude higher — possibly due to intermediate data structures. There may still be room for optimization here. That said, it no longer seems like a memory leak to me. I believe the proper long-term solution would be to introduce pagination in the /jobs handler.
Glad I could help! Pagination indeed sounds like a more scalable solution for long term Hope Schedmd agrees and maybe we see it in later versions :)
