Ticket 1732

Hi,
my name is Herbert Mehlhose and I'm working on the SLURM configuration at PIK Potsdam. I did already compile slurm (14.11.7 currently, will compile 14.11.8 if you recommend this). As recommended I compiled munge and got this stuff basically working, so submitting simple jobs into partitions with sbatch is basically working.
The setup:
Cluster with 318 compute nodes SLES11 SP3 (stateless boot)
Filesystem GPFS
Provisioning is done via xCAT.
What I currently have implemented: xCAT installs all slurm and munge rpms required, and within an xCAT postscript the correct permissions etc. are set to the required directories. Basically: slurmd and slurmctld come up after a new installation or stateless rebooot.
I have been busy with many things during setup now, so my colleague Holger Holthoff already filed a ticked #1731 - this contains many questions the customer asked me - and this is quite urgent.

I want to file a new ticket here, regarding just one simple question. I think it is a good way to ask simple questions in single tickets, so that we have a short reference and can close these.

My current setup uses xCAT syncfiles to distribute the munge key and the slurm.conf. 
For the munge key, this is ok, as this is absolutely static.
For the slurm.conf: I would like to include this into a shared filesystem (here: GPFS), so that in case of any changes in configuration files, we do not have to sync these contents via xcat (which does rsync) to the nodes. This could result nodes not receiving the updates for whatever reasons. So using a shared FS will just require a service slurm reconfig (or maybe restart) to have changes available. And having a consistent configuration across the cluster is vital to avoid problems, that's what I get from the documentation.

So I tried this today and found, that there are several ways possible to change the path to the config:
a) at build time: DEFAULT_SLURM_CONF
b) at execution time:  by setting the SLURM_CONF environment variable
c) -f option at daemon startup

Option a) sounds interesting - might test that with next version build
Option b) - simply did not work - checking the /etc/init.d/slurm script shows me, that it checks for fixed /etc/slurm/ path
Option c) also does not work with init script.

I found, that running /usr/sbin/slurmd will accept the env variable and the -f option. But I want to use the init script, because this seems automatically to distinguish from the information in the configuration, which daemon has to be started. At least in my setup, on the controller nodes, it starts slurmctld and on the compute nodes it starts slurmd - it must get this information from the configuration file, but if it can't read this if it is located somewhere else, it is quite useless for my shared setup.

What I did now: just create a symlink "ln -s /p/system/slurm /etc/slurm" within my postscript. This works fine.

But: is this the recommended way to do it? Why does the init script not reflect the environment variable? Using /usr/sbin/slurmXX will require me to check manually, which role the node has.

BTW: next step I want to do is to be prepared for slurmctld failover - this means, that I'll go to include "StateSaveLocation" into the shared GPFS as well. But as said - another story and I'll open new ticket for this topic.

Thanks for any tips for good practice how to start the daemons (init script vs. direct call etc...)

Best regards,
Herbert