We have a project that would like to run over 100000 jobs. Currently they submit jobs every 0.1 sec and Slurm controller is OK, but as soon as they try submitting jobs every 0.05 sec controller comes to a halt. What should be an ideal job submission time limits? What is a best way to sumbit 100000 jobs, which would not impact a controller?
(In reply to NASA JSC Aerolab from comment #0) > We have a project that would like to run over 100000 jobs. Currently they > submit jobs every 0.1 sec and Slurm controller is OK, but as soon as they > try submitting jobs every 0.05 sec controller comes to a halt. What should > be an ideal job submission time limits? What is a best way to sumbit 100000 > jobs, which would not impact a controller? Hi, There are many parameters we can tune. I'd like to start analyzing what you currently have set in slurm.conf and from this I will see if there's a valid proposal for you. Also, are these single bursts of 100.000 jobs? or is it continue? How many jobs are usually in the queue? How many jobs do you allow to be started at a time? Is throughput important or just want to be able to submit all these jobs and let the scheduler do its job more "slowly"? There's usually a compromise between responsiveness to RPCs (commands, communications, and so on) and scheduling work. Please, send your slurm.conf and if possible define a little bit more the workflow.
Created attachment 9572 [details] slurm config file.
Jobs are submitted continuously, but ideally we would like to submit 100000 jobs and let slurm take care of it. We don't have many limits in place at the moment as you will see in our slurm.conf file. Workflow: File manipulation -> computation which generate files -> file manipulation which takes place in parallel.
(In reply to NASA JSC Aerolab from comment #5) > Jobs are submitted continuously, but ideally we would like to submit 100000 > jobs and let slurm take care of it. We don't have many limits in place at > the moment as you will see in our slurm.conf file. > > Workflow: File manipulation -> computation which generate files -> file > manipulation which takes place in parallel. Well, we will try to set some initial parameters and see how it works. Tuning schedule is normally a process that require many iterations until we get the correct settings. We will start by backfill parameters, and continue with builtin scheduler ones. I see you just have 41 nodes with 56 threads each. This is a small system, so it won't be possible to run 100000 jobs at a time. That means that we do not need to try to schedule a lot of jobs because anyway they won't fit in the nodes. Let's say that each job can use one single thread. There are 56*41=2296 CPUs available which means that at most, there will be 2296 jobs running in the system. That would be true if you hadn't the "Oversubscribe=YES" in the partition, in any case we will assume this value for now. I would say then to set the max queue depth to 500, with this setting you will see a big percentage of jobs tried to be/being scheduled at each iteration, with a compromise with responsiveness. If responsiveness is still not good, we can decrease this value. bf_max_job_test=500 We can limit the amount of time the backfill scheduler can spend in one cycle even if maximum job count has not been reached. This is a control for improving responsiveness. For now we are not changing the default value (same as bf_interval), but we can take this one into account for the future. bf_max_time= Since we are potentially cutting the scheduling when it is in the middle of a cycle, lets also add the bf_continue parameter. bf_continue To limit the number of seconds between backfill iterations we can set bf_interval. Default is set at 30 seconds, so we can leave it as is: bf_interval= Also, we can limit the scheduler resolution when looking and planning for the future. Higher values gives better responsiveness, but scheduling can be a bit less precise, which I guess it is not so important. This is affected by bf_interval which we could read as: Looking bf_window minutes in the future with bf_resolution seconds resolution. Your bf_window is not big as you will see below, but in any case let's set 5 minutes just in case. If everything works fine you can decrease this parameter to achieve more precise scheduling times. bf_resolution=300 Then we can instruct the scheduler to just look into the future for a limited number of minutes. This value has to be at least as long as the highest allowed time limit. I see your partition with DefaultTime=04:00:00 MaxTime=8:00:00. So lets put here 8 hours. If everything works fine, you can increase this parameter. Take in mind bf_resolution too: bf_window=480 We then have bf_yield_interval and bf_yield_sleep. These are, each X seconds release the locks to permit RPCs and other operations to be processed, and for how Y seconds to yield these locks, respectively. Default is 2 seconds and 0.5 seconds. If you continue seeing responsiveness issues, you can make the scheduler to "stop" more frequently decreasing bf_yield_interval, or let it sleep for more time with bf_yield_sleep. We will leave them with default values, but if you continue seeing responsiveness problems, use this parameters. bf_yield_interval= bf_yield_sleep= Added to the previous two, we can also change max_rpc_cnt. This one will instruct backfill to defer work when there are too many RPCs pending. Also, the time that the scheduler will sleep will depend on this parameter. It will sleep at least as the max_rpc_cnt is reduced by 10 times. I will put here a value of 100, since there can be at most 256 pending operations to be served at a time (equivalent to MAX_SERVER_THREADS). This parameter should help responsiveness: max_rpc_cnt=100 Now for the builtin scheduler: The equivalent of bf_max_job_test but for the builtin scheduler, let's keep the default of 100 here, because this affects the number of jobs to try to be scheduled at submit time or when a job finishes: default_queue_depth= If submitting thousands of jobs still blocks ctld, we can evaluate the drastic 'defer' parameter later. Finally one more than can help us with your responsiveness. Every job submission the builtin scheduler will start trying to schedule, by default, at most every 1 second interval. Since you are submitting thousands of jobs per second, the scheduler will be running all the time. Let's make it to not try to schedule jobs at as short intervals by increasing from 1 second to 2 seconds. sched_min_interval=2000000 Others parameters to consider if after a couple of iterations we still doesn't achieve desired performance: max_sched_time=# sched_interval To summarize, let's try to set this line in slurm.conf, and restart slurmctld, then run a reconfigure: SchedulerParameters=bf_max_job_test=500,bf_resolution=300,bf_window=480,max_rpc_cnt=100,bf_continue,sched_min_interval=2000000 These parameters are for the future: bf_interval= bf_max_time= bf_yield_interval= bf_yield_sleep= default_queue_depth= Tell me what improvements you see and attach here some sdiag outputs captured in a periodic basis (each 10 seconds during 5 minutes when submitting 100000 jobs).
I'm reviewing your recommendations and will inform you after we implement and run some jobs with it. According to the following documentation, array can be used to distribute millions of jobs in milliseconds. Do you think it's possible with compute resource we have. https://slurm.schedmd.com/job_array.html
(In reply to NASA JSC Aerolab from comment #7) > I'm reviewing your recommendations and will inform you after we implement > and run some jobs with it. > > According to the following documentation, array can be used to distribute > millions > of jobs in milliseconds. Do you think it's possible with compute resource we > have. > > https://slurm.schedmd.com/job_array.html Yes, indeed. It is a good practice and we recommend so. If you are able to use job arrays maybe your problem will be much less complicated. I didn't know if this many jobs were from several users or just one, so I assumed you wanted just scheduler tunning. In any case, you can implement the proposed parameters, and then use job arrays.
Only one user was submitting all jobs. Currently MaxArraySize is set to 2500, In past a single user tried to submit around 100K jobs, but it brought machine down to halt due to lack of Memory. How much RAM does Slurm recommend if we increase MaxArraySize to 10000 and MaxJobCount=100000. Should a user submit 10 different jobs to submit 100K jobs? Should there be a delay of a second between each job submission?
(In reply to NASA JSC Aerolab from comment #9) > Only one user was submitting all jobs. > Currently MaxArraySize is set to 2500, In past a single user tried to submit > around 100K jobs, but it brought machine down to halt due to lack of Memory. > How much RAM does Slurm recommend if we increase MaxArraySize to 10000 and > MaxJobCount=100000. > > Should a user submit 10 different jobs to submit 100K jobs? > Should there be a delay of a second between each job submission? When a job array is submitted just one "object" is created in memory with the metadata of the array until it is modified or initiated, so there will be no RAM issues at that time. Nevertheless performance can be impacted due to scheduling. One key here is to limit the number of jobs to be scheduled at a time. I cannot say a number for RAM, it will depend on how many jobs you will start or modify at a given time and the scheduler pressure. I think that with the provided parameters plus something like you propose: MaxArraySize=10000 MaxJobCount=100000 and maybe setting a MaxSubmitJobs if users are misbehaving, it should be good enough. You can even increase both values, but I'd recommend to try it step by step and monitor the system resources. This is an example on my laptop submitting a job array of 30000 tasks, almost no memory consumption since I just schedule/initiate a few jobs at a time: [lipi@llagosti 18.08]$ free -m total used free shared buff/cache available Mem: 7858 3508 734 787 3614 3290 Swap: 1535 2 1533 [lipi@llagosti 18.08]$ sbatch -N1 -a 1-29999 --wrap='sleep 1' Submitted batch job 6561 [lipi@llagosti 18.08]$ free -m total used free shared buff/cache available Mem: 7858 3548 685 794 3623 3242 Swap: 1535 2 1533 [lipi@llagosti 18.08]$ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 6544_1 debug wrap lipi CG 0:02 1 gamba1 6544_2 debug wrap lipi CG 0:03 1 gamba1 6544_3 debug wrap lipi CG 0:04 1 gamba1 6544_4 debug wrap lipi CG 0:04 1 gamba1 6561_[42-29999] debug wrap lipi PD 0:00 1 (Resources) 6561_41 debug wrap lipi R 0:00 1 gamba2 6561_39 debug wrap lipi R 0:01 1 gamba3 6561_40 debug wrap lipi R 0:01 1 gamba3 6561_29 debug wrap lipi R 0:02 1 gamba3 6561_30 debug wrap lipi R 0:02 1 gamba3 6561_31 debug wrap lipi R 0:02 1 gamba2 6561_32 debug wrap lipi R 0:02 1 gamba2 6561_33 debug wrap lipi R 0:02 1 gamba2 6561_34 debug wrap lipi R 0:02 1 gamba4 6561_35 debug wrap lipi R 0:02 1 gamba4 6561_36 debug wrap lipi R 0:02 1 gamba4 6561_37 debug wrap lipi R 0:02 1 gamba4
That's a synthetic test launching 90.000 jobs within 3 arrays and starting a few of them. Almost no RAM consumption and launched at the same time. Submit a fourth 30000 job would fail because my MaxJobCount is 100000 and in total it would've required 120.000 jobs. [lipi@llagosti 18.08]$ free -m total used free shared buff/cache available Mem: 7858 3654 2736 593 1467 3353 Swap: 1535 3 1532 [lipi@llagosti 18.08]$ cat /tmp/test.sh #!/bin/bash sbatch -N1 -a 1-29999 --wrap='sleep 1' && sbatch -N1 -a 1-29999 --wrap='sleep 1' && sbatch -N1 -a 1-29999 --wrap='sleep 1' [lipi@llagosti 18.08]$ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) [lipi@llagosti 18.08]$ time /tmp/test.sh Submitted batch job 9093 Submitted batch job 9094 Submitted batch job 9095 real 0m0,089s user 0m0,032s sys 0m0,028s [lipi@llagosti 18.08]$ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 9093_7 debug wrap lipi CG 0:01 1 gamba3 9093_9 debug wrap lipi CG 0:01 1 gamba4 9093_12 debug wrap lipi CG 0:01 1 gamba4 9094_[1-29999] debug wrap lipi PD 0:00 1 (Priority) 9095_[1-29999] debug wrap lipi PD 0:00 1 (Priority) 9093_[15-29999] debug wrap lipi PD 0:00 1 (Resources) 9093_13 debug wrap lipi R 0:00 1 gamba2 9093_14 debug wrap lipi R 0:00 1 gamba3 [lipi@llagosti 18.08]$ free -m total used free shared buff/cache available Mem: 7858 3701 2689 583 1466 3316 Swap: 1535 3 1532 [lipi@llagosti 18.08]$ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 9094_[1-29999] debug wrap lipi PD 0:00 1 (Priority) 9095_[1-29999] debug wrap lipi PD 0:00 1 (Priority) 9093_[51-29999] debug wrap lipi PD 0:00 1 (Resources) 9093_49 debug wrap lipi R 0:00 1 gamba2 9093_50 debug wrap lipi R 0:00 1 gamba3 9093_39 debug wrap lipi R 0:01 1 gamba2 9093_40 debug wrap lipi R 0:01 1 gamba2 9093_41 debug wrap lipi R 0:01 1 gamba2 9093_42 debug wrap lipi R 0:01 1 gamba3 9093_43 debug wrap lipi R 0:01 1 gamba3 9093_44 debug wrap lipi R 0:01 1 gamba3 9093_45 debug wrap lipi R 0:01 1 gamba4 9093_46 debug wrap lipi R 0:01 1 gamba4 9093_47 debug wrap lipi R 0:01 1 gamba4 9093_48 debug wrap lipi R 0:01 1 gamba4 [lipi@llagosti 18.08]$ free -m total used free shared buff/cache available Mem: 7858 3682 2706 585 1469 3333 Swap: 1535 3 1532
Hi, have you applied the recommendations or tried with arrays? Is the scheduler performing better?
Only change I have made is on array size as we discussed and I think we are good for right now. Thank you for your assistance.
Ok, I am marking the issue as Infogiven then. Please reopen if you still see issues even after implementing our recommendations. Regards
