This is the error messages from slurmdbd.log

[2022-07-14T03:45:02.425] error: ***********************************************
[2022-07-14T03:45:02.425] error: Can not recover archive file, incompatible version, got 9728 need <= 9472
[2022-07-14T03:45:02.425] error: ***********************************************

Hi Ke,

I'm afraid there isn't much we can do to make an older version of slurmdbd read an archive file from a newer version.  There are some changes to tables between these versions, so the older slurmdbd can't load the data because there isn't a corresponding column in the database it maintains.  The solution for this would be to upgrade the archive instance of slurmdbd to version 22.05.  You can reference the upgrade guide for information on how to perform an upgrade like this.

https://slurm.schedmd.com/quickstart_admin.html#upgrade

Let me know if you have any questions about this.

Thanks,
Ben

Hi Ben,

In this case, how to deal with the Slurm clusters with different version ?

Because we can't upgrade all clusters at the same time. Particular the largest cluster, it takes a lot of testing and planning.

Newer versions of slurmdbd should be able to read archive information from older instances of slurmdbd.  For example, if your clusters are on 21.08 and 22.05 you can upgrade your archive server to 22.05 and have things work as expected.

It is possible that there might be problems reading archive files from older slurmdbd instances if the difference is great enough.  Which versions are you going to be importing data from?  

Thanks,
Ben

Hi Ben,

We are upgrading our cluster from 21.08 to 22.05. So we are running 21.08 and 22.05.

I think I can upgrade the archive slurmdbd to 22.05. But how to upgrade the tables? Since the archive database contains tables from multiple clusters.
How to only update the tables only for cluster that is upgraded?

Thanks,

Ke

You're right that if you upgrade slurmdbd on your archive server it will update the tables for both clusters.  If you are using the archive server to generate reports with sreport or pulling data with sacct then I wouldn't expect there to be any noticeable difference with the updated database.  If you are running direct mysql queries on the tables then you may have to update the query to account for the changes in the tables.  

You should be able to get a good feel for how any differences would look by running the same reports/queries on your production systems with the different versions of slurmdbd.  If you don't see any difference there, then you shouldn't notice any difference when running the same reports/queries against the data on your archive server.

Thanks,
Ben

Hi Ke,

Do you have any additional questions about upgrading your archive instance of slurmdbd?  Let me know if you still need help with this or if the ticket is ok to close.

Thanks,
Ben

Hi Ben,

I tested upgrading the archive instance. It seems working as expected. Please close the ticket. 

Thanks,

Ke

I'm glad to hear that went smoothly.  Let us know if there's anything else we can do to help.

Thanks,
Ben