Ticket 14880

Is a question specific to scaleout <https://github.com/naterini/docker-scale-out> or Slurm in general?

Sorry, Slurm in general.  I've found that increasing ports and inceasing Munge threads has helped in my testing but was wondering if SchedMD had a rule-of-thumb or formula for the amount of Slurmctld ports for X amount of compute nodes.  

I am not using docker or anything like that, I was not aware that was a SchedMD product.  I assumed scaleout was for scaling in general.  

Thanks!

(In reply to Nick McCollum from comment #2)
> Sorry, Slurm in general.  I've found that increasing ports and inceasing
> Munge threads has helped in my testing but was wondering if SchedMD had a
> rule-of-thumb or formula for the amount of Slurmctld ports for X amount of
> compute nodes.
Any modern Linux kernel should make any performance increases from multiple ports vs one negligible at best. If there is a difference in performance, then the sysctl net parameters need some tweaking. All the connections go to the same code in Slurm.

The number of munge threads does matter heavily for bursts.

Can you please provide more information on how you are testing and what is being tested specifically?

> I am not using docker or anything like that, I was not aware that was a
> SchedMD product.  I assumed scaleout was for scaling in general.  
We use it for training. It allows anyone to run a Slurm cluster on their laptop and test changes without affecting their prod systems. The name scaleout is from the fact it can easily scale out the number of nodes running the test cluster.

I feel like increasing from SlurmctldPort=6810-6817 to SlurmctldPort=6800-6817 has made an improvement in my 1200 node test cluster.

(In reply to Nick McCollum from comment #4)
> I feel like increasing from SlurmctldPort=6810-6817 to
> SlurmctldPort=6800-6817 has made an improvement in my 1200 node test cluster.

There may be something else involved that is being helped by the extra listening ports. On the host running slurmctld, please call:
> sudo netstat -tulpn
> ip r
> ip a

Please respond to the request in comment#6 when convenient, and we can continue debugging.

Thanks,
--Nate