Hi Paul. From/to which versions did you upgrade? It looks like client commands and slurmctld differ one to each other more than 2 minor releases:


[2017-02-21T15:19:04.368] error: slurm_receive_msg: Incompatible versions of client and server code
[2017-02-21T15:19:04.372] error: slurm_receive_msg: Invalid Protocol Version 7680 from uid=-1 at 172.16.250.121:43092

Can you check the current version of all your components?

Remember that Slurm daemons will support RPCs and state files from the two previous minor releases (e.g. a version 16.05.x SlurmDBD will support slurmctld daemons and commands with a version of 16.05.x, 15.08.x or 14.11.x).

14.11.10 to 16.05.7

Again this worked fine prior, but failed in this cluster.

I'm going to state the obvious...  BUT when it failed to read the state file, wouldn't it be a good time to exit and not kill every job on the cluster?

This is obviously going to be a massive mess for our users to clean up.

What we need to know (and hopefully quickly) is... can we recover?  We only have a backup slurmdb and NOT the state files (unless slurp keeps backups somewhere).

If we can't recover... then we might as well get on with life, restart slurm on the new version and clean up.

Slurm state files are located under StateSaveLocation, and Slurm doesn't make backups of it. I think you won't be able to recover the state previous to the upgrade. Can you please attach the full slurmctld.log? Did you upgrade slurmdbd first?

Created attachment 4076 [details]
slurmctrld.log

Yes slurmdbd was upgraded first. Please see the attached log.

[2017-02-21T15:19:06.960] slurmctld version 16.05.7 started on cluster perth
...
[2017-02-21T15:19:11.296] error: we don't have select plugin type 4096
[2017-02-21T15:19:11.296] error: select_g_select_jobinfo_unpack: unpack error
[2017-02-21T15:19:11.296] error: Incomplete job record
[2017-02-21T15:19:11.296] error: Incomplete job state save file
[2017-02-21T15:19:11.296] Recovered information about 0 jobs
[2017-02-21T15:19:11.297] cons_res: select_p_node_init
[2017-02-21T15:19:11.297] cons_res: preparing for 150 partitions
[2017-02-21T15:19:11.298] Purged files for defunct batch job 23083128
...

Have you recently changed slurm.conf switching SelectType? 

[2017-02-21T15:19:11.296] error: we don't have select plugin type 4096

If so, it may be possible that the controller is using cons_res, but the compute node was not restarted and is still using a different select type. I also see tons of messages related to different slurm.conf files across the cluster. Can you check that as well as the version of all slurmd's in the cluster, slurmdbd version, slurmctld and commands versions?

We have not had any changes with plugins from version 14.11.10 to version 16.05.7. 

All of the slurm.conf version errors were due to the controller restarting into the new version with MemSpecLimit changing for all nodes. The clients were yet to be restarted to be configured with the new config file (next step after restarting the controller).

How did you install? was it with rpm and if so did you install it with the Slurm rpm plugins? can you check the dates of your select_*.so files? can you attach your slurm.conf as well? where's PluginDir pointing to?

Also, in which order did you upgrade the components? you said slurmdbd was first. And then? Perhaps did you upgrade first slurmd's before slurmctld? 

Have you followed the safe upgrade procedure accessible in this link?

https://slurm.schedmd.com/quickstart_admin.html

This error doesn't make much sense since I don't see we have any plugin type with plugin id 4096, so I don't know if you had a state file older than expected or corrupted.

[2017-02-21T15:19:11.296] error: we don't have select plugin type 4096

If we had a copy of the SlurmStateDir as it is recommended we could have something to work of off, but we don't so it's difficult to tell what exactly happened. From now on I'd recommend go and get a stable production cluster, get all the components running with the same version and ensure there's just one version of the slurm.conf across the whole cluster.

Hi Alejandro,

I believe I have found the cause.

We link all of our daemons to a directory, latest, for slurm on a network file system. "latest" then points to the current version. For slurmdbd, I manually pointed the daemon to the new versions directory, but when I did slurmctld I changed latest to point to the newer version. I have done this before in the test environment without issue, but in this situation, it seems the errors from the clients not being pointed to the proper version (for the old bin, lib..etc) were more than slurm could handle. 

With that said, I am concerned with slurm continuing after failing to save job states and purging files. I would think it would require a config setting to override state saving on daemon restart and if not set fails if saving fails. Or something of the kind. 

In this situation, we have moved forward with the new version, requiring resubmission of all jobs. Measures will be put in place as a fail safe moving forward.

I do appreciate the quick response. Any idea if any of this can be added to avoid this happening again?

Thanks,
Paul

(In reply to paull from comment #13)
> Hi Alejandro,
> 
> I believe I have found the cause.
> 
> We link all of our daemons to a directory, latest, for slurm on a network
> file system. "latest" then points to the current version. For slurmdbd, I
> manually pointed the daemon to the new versions directory, but when I did
> slurmctld I changed latest to point to the newer version. I have done this
> before in the test environment without issue, but in this situation, it
> seems the errors from the clients not being pointed to the proper version
> (for the old bin, lib..etc) were more than slurm could handle. 
> 
> With that said, I am concerned with slurm continuing after failing to save
> job states and purging files. I would think it would require a config
> setting to override state saving on daemon restart and if not set fails if
> saving fails. Or something of the kind. 
> 
> In this situation, we have moved forward with the new version, requiring
> resubmission of all jobs. Measures will be put in place as a fail safe
> moving forward.
> 
> I do appreciate the quick response. Any idea if any of this can be added to
> avoid this happening again?
> 
> Thanks,
> Paul

Well, there's already a backup of the state files within the StateSaveLocation, they are marked as .old, and get over written each restart in a rotation fashion. What I meant in comment #5 is that Slurm doesn't backup the whole StateSaveLocation somewhere else, but still there's a backup .old copy in that directory. If you still preserve the .old files from before the upgrade attempt, we'd be interested in analyzing them, specially for this error:

[2017-02-21T15:19:11.296] error: we don't have select plugin type 4096

I looked at the .old file and due to the server being restarted a couple of times, it contained the same info as the normal state file.

BUT lets be really clear here.

This is still a massive bug in slurm.  Slurm should NOT basically delete every job on the system (running and queued) if it can't read the state file... it should EXIT and tell the user the issue and let the user delete the files if they want to... or find some other solution.

Apologies for not responding faster; Alex is out on vacation and this is my fault for not catching this yesterday.

There are two separate issues here, and I'm filing a separate bug to cover part of it:

1) The upgrade was done out of order, resulting in running jobs being cancelled. Unfortunately, there isn't much we can do to prevent this due to the way the protocols are designed currently - to handle this properly would require some significant architectural changes which I do not feel are worth the associated complexity at this time.

2) slurmctld could not read the job_state file at startup. I would still like to understand the root cause of this - the "4096" value that was read as a select plugin type does not correspond to anything within the slurm source code, and either indicates you may have had some local patch running that modified the state format (which we haven't received confirmation on either way yet, comment #1 on bug 3484), or that the state file had been corrupted somehow.

If the state file was corrupted, as it appears to have been, there would be no way to recover the system state. If this was due to a local modification, it may have been possible to proceed, but would be outside the scope of our support.

However, if you did not take any backups (as recommended in step #8 https://slurm.schedmd.com/quickstart_admin.html#upgrade here) before, and do not have a copy of that file I do not think we'll be able to reproduce this, and should thus focus our efforts on preventing such an event in the future.

I agree that the startup code should be modified to fail-fast, and rather than proceed to cancel all jobs, refuse to start instead with some warnings as to why the cluster cannot be brought up properly, and how to override that limit. I've opened bug 3502 to track this issue, and will make sure this is reviewed and finished before the next point release for 16.05 and 17.02 are issued.

Is there anything else we can address on this specific issue, or can I close this and focus our efforts on bug 3502?

- Tim

I'm happy to close.

Paul has the upgrade procedure sorted and has done all our offices now.

Thanks.

Marking resolved/infogiven, and moving on to bug 3484 to cover mitigating the impact of unloadable state files.

We were just bit by this again (apparently changing SelectType torches the entire job state?), and I don't seem to have access to bug 3502.  Is it intentionally private?