We have upgraded our slurmdbd server to 23.02.4 while the controller and the rest of the cluster is still running 22.05.9. On the 23.02.4 all user commands seem to fail when querying the slurmctld, for example: $ sinfo slurm_load_partitions: Unexpected message received $ squeue slurm_load_jobs error: Unexpected message received $ scontrol show partitions slurm_load_partitions error: Unexpected message received These commands work correctly on the 22.05.9 nodes. Can you please help us getting the 23.02.4 commands to work with the 22.05.9 controller? Thanks, Ole
Created attachment 31749 [details] slurm.conf from the 22.05.9 controller
Hello Ole, Unfortunately, newer client commands can not be used with older versions. The RPC versioning will be something the older slurmctld does not understand or know how to handle. You will have to either upgrade the slurmctld or use older client commands with your current slurmctld version. https://slurm.schedmd.com/quickstart_admin.html#upgrade > The slurmctld daemon must also be upgraded before or at the same time as the > slurmd daemons on the compute nodes. Generally, upgrading Slurm on all of the > login and compute nodes is recommended, although rolling upgrades are also > possible (i.e. upgrading the head node(s) first then upgrading the compute and > login nodes later at various times). Also see the note above about reverse > compatibility. Regards, --Megan
Hi Megan, Thanks for the clarification: (In reply to Megan Dahl from comment #3) > Unfortunately, newer client commands can not be used with older versions. > The RPC versioning will be something the older slurmctld does not understand > or know how to handle. You will have to either upgrade the slurmctld or use > older client commands with your current slurmctld version. Unfortunately, this restriction seems to be undocumented: The inability of new version user commands to communicate with an older version of slurmctld. The upgrade page currently doesn't cover this case: > https://slurm.schedmd.com/quickstart_admin.html#upgrade Could you kindly add documentation of the lack of backwards compatibility of user commands with an older version slurmctld? Thanks, Ole
Hi Ole, I will work on including the restriction into the documentation. I will let you know when it has been added. Thanks, --Megan
Hello Ole, The clarification that newer client commands can not be used with older versions has been added to the documentation in the following commit. commit ce51a8fa6ecdbaef2fc7e2077db3daff934fb9cd Author: Megan Dahl <megan@schedmd.com> Date: Mon Aug 14 13:48:05 2023 -0600 Docs - Indicate that slurmctld must be upgraded before client commands Bug 17418 This will be available in 23.02.6. Regards, --Megan
Hi Megan, (In reply to Megan Dahl from comment #8) > The clarification that newer client commands can not be used with older > versions has been added to the documentation in the following commit. > > commit ce51a8fa6ecdbaef2fc7e2077db3daff934fb9cd > Author: Megan Dahl <megan@schedmd.com> > Date: Mon Aug 14 13:48:05 2023 -0600 > > Docs - Indicate that slurmctld must be upgraded before client commands > > Bug 17418 > > This will be available in 23.02.6. Thanks a lot! Ole