Summary: | slurm_load_partitions: Unexpected message received | ||
---|---|---|---|
Product: | Slurm | Reporter: | Ole.H.Nielsen <Ole.H.Nielsen> |
Component: | User Commands | Assignee: | Megan Dahl <megan> |
Status: | RESOLVED FIXED | QA Contact: | Ben Roberts <ben> |
Severity: | 4 - Minor Issue | ||
Priority: | --- | ||
Version: | 23.02.4 | ||
Hardware: | Linux | ||
OS: | Linux | ||
Site: | DTU Physics | Alineos Sites: | --- |
Atos/Eviden Sites: | --- | Confidential Site: | --- |
Coreweave sites: | --- | Cray Sites: | --- |
DS9 clusters: | --- | HPCnow Sites: | --- |
HPE Sites: | --- | IBM Sites: | --- |
NOAA SIte: | --- | NoveTech Sites: | --- |
Nvidia HWinf-CS Sites: | --- | OCF Sites: | --- |
Recursion Pharma Sites: | --- | SFW Sites: | --- |
SNIC sites: | --- | Linux Distro: | --- |
Machine Name: | CLE Version: | ||
Version Fixed: | 23.02.5 | Target Release: | --- |
DevPrio: | --- | Emory-Cloud Sites: | --- |
Attachments: | slurm.conf from the 22.05.9 controller |
Description
Ole.H.Nielsen@fysik.dtu.dk
2023-08-14 02:15:45 MDT
Created attachment 31749 [details]
slurm.conf from the 22.05.9 controller
Hello Ole, Unfortunately, newer client commands can not be used with older versions. The RPC versioning will be something the older slurmctld does not understand or know how to handle. You will have to either upgrade the slurmctld or use older client commands with your current slurmctld version. https://slurm.schedmd.com/quickstart_admin.html#upgrade > The slurmctld daemon must also be upgraded before or at the same time as the > slurmd daemons on the compute nodes. Generally, upgrading Slurm on all of the > login and compute nodes is recommended, although rolling upgrades are also > possible (i.e. upgrading the head node(s) first then upgrading the compute and > login nodes later at various times). Also see the note above about reverse > compatibility. Regards, --Megan Hi Megan, Thanks for the clarification: (In reply to Megan Dahl from comment #3) > Unfortunately, newer client commands can not be used with older versions. > The RPC versioning will be something the older slurmctld does not understand > or know how to handle. You will have to either upgrade the slurmctld or use > older client commands with your current slurmctld version. Unfortunately, this restriction seems to be undocumented: The inability of new version user commands to communicate with an older version of slurmctld. The upgrade page currently doesn't cover this case: > https://slurm.schedmd.com/quickstart_admin.html#upgrade Could you kindly add documentation of the lack of backwards compatibility of user commands with an older version slurmctld? Thanks, Ole Hi Ole, I will work on including the restriction into the documentation. I will let you know when it has been added. Thanks, --Megan Hello Ole, The clarification that newer client commands can not be used with older versions has been added to the documentation in the following commit. commit ce51a8fa6ecdbaef2fc7e2077db3daff934fb9cd Author: Megan Dahl <megan@schedmd.com> Date: Mon Aug 14 13:48:05 2023 -0600 Docs - Indicate that slurmctld must be upgraded before client commands Bug 17418 This will be available in 23.02.6. Regards, --Megan Hi Megan, (In reply to Megan Dahl from comment #8) > The clarification that newer client commands can not be used with older > versions has been added to the documentation in the following commit. > > commit ce51a8fa6ecdbaef2fc7e2077db3daff934fb9cd > Author: Megan Dahl <megan@schedmd.com> > Date: Mon Aug 14 13:48:05 2023 -0600 > > Docs - Indicate that slurmctld must be upgraded before client commands > > Bug 17418 > > This will be available in 23.02.6. Thanks a lot! Ole |