Hi, We are installing Slurm 18.08.7 on a large configuration (around 1800 compute nodes). Submitted jobs can either be some large jobs requesting 1500 nodes or a burst of small jobs. Commands like squeue or sinfo frequently return this error : Socket timed out on send/recv operation, especially when large jobs are launched or terminated. MessageTimeout has already been increased to 20 ; we understand that we may increase this value up to 90 but from users perspective, waiting for the result of squeue or sinfo for more than 20 seconds can be a real problem. We are also aware that the tuning of Slurm can be tricky, so could you please help us to cope with this problem ? Best regards, Philippe COLLINET
Hi Philippe, Will you kindly attach: - a copy of your config files - slurmctld.log - output of sdiag We keep track in an internal repo a copy of our customers configuration, so we don't need to ask for config attachments on every single bug. Also, please remember to word any password options like in i.e. slurmdbd.conf. Thanks
Hello, We setup the debug mode and it helped US. We found that the address resolution of the nodes running slurm commands was incorrect and had to wait for the second address. Whence this issue corrected, we do not suffer Socket timed out on send/recv operation. Thanks for your help. We can close the Bug. Best regards, Philippe Collinet
Hi Philippe, These are good news. Thanks for the explanations. I'm closing this ticket as infogiven (although it has been you who give the info to us ;-) Please let us know if you need further support, Albert
