SchedMD - Slurm Support – Full Text Ticket Listing
- Home
- | New
- | Browse
- | Search
- | [?]
- | Reports
- | Help
- | New Account
- | Log In
- | Forgot Password
| Summary: | Slurm install with different SLURM_PREFIX no longer works inside a job run with a different Slurm install | ||
|---|---|---|---|
| Product: | Slurm | Reporter: | Chris Samuel (NERSC) <csamuel> |
| Component: | User Commands | Assignee: | Marcin Stolarek <cinek> |
| Status: | RESOLVED INFOGIVEN | QA Contact: | |
| Severity: | 3 - Medium Impact | ||
| Priority: | --- | CC: | cinek, dmjacobsen, tim |
| Version: | 20.02.3 | ||
| Hardware: | Linux | ||
| OS: | Linux | ||
| Site: | NERSC | Slinky Site: | --- |
| Alineos Sites: | --- | Atos/Eviden Sites: | --- |
| Confidential Site: | --- | Coreweave sites: | --- |
| Cray Sites: | --- | DS9 clusters: | --- |
| Google sites: | --- | HPCnow Sites: | --- |
| HPE Sites: | --- | IBM Sites: | --- |
| NOAA SIte: | --- | NoveTech Sites: | --- |
| Nvidia HWinf-CS Sites: | --- | OCF Sites: | --- |
| Recursion Pharma Sites: | --- | SFW Sites: | --- |
| SNIC sites: | --- | Tzag Elita Sites: | --- |
| Linux Distro: | --- | Machine Name: | |
| CLE Version: | Version Fixed: | ||
| Target Release: | --- | DevPrio: | --- |
| Emory-Cloud Sites: | --- | ||
|
Description
Chris Samuel (NERSC)
2020-06-05 18:58:15 MDT
This is all the esslurm module does: modulecsamuel@gert01:~> module show esslurm ------------------------------------------------------------------- /usr/common/software/modulefiles/esslurm: module-whatis NERSC es Slurm binary executable module prepend-path PATH /opt/esslurm/bin prepend-path LD_LIBRARY_PATH /opt/esslurm/lib64 ------------------------------------------------------------------- Chris,
This looks like a side-effect of the config-less approach introduced in Slurm 20.02. Specifically, to make sure that commands make use of the same cached version of slurm.conf the SLURM_CONF environment variable gets set.
Although environment modules are outside of SchedMD expertise I think that adding:
>set OLD_SLURM_CONF [ getenv SLURM_CONF]
>unsetenv SLURM_CONF $OLD_SLURM_CONF
to esslurm module file should work for you. The goal is to unset SLURM_CONF when loading the module, but set it back to the previous value if someone will unload the module attempting to use local build - creating a step within existing allocation on gerty after submission to xfer.
Let me know if that worked.
cheers,
Marcin
We also have a number of use-cases where calling the executable by absolute path, (e.g., /opt/esslurm/bin/scontrol) needs to work, so I do think that we need to assure that an executable can find its own configuration without relying on the environment. Doug, Just to make sure we're on the same page. Calling the binary within a clean environment (without SLURM_CONF variable set) will work as you expect and use slurm.conf from the binary default location. The issue here is calling a Slurm binary from the existing allocation against other configuration than initial salloc/sbatch/srun. The simple workaround that should work for this case is the use of wrapper that unsets SLURM_CONF(or sets appropriately) before calling a Slurm command. What do you think? cheers, Marcin Hi Marcin, Whilst setting up wrappers is technically possible it is a bit of a hack to try and work around something that used to work. What I'm wondering, looking at _establish_config_source() in src/common/read_config.c, is that it looks like it might be able to fix this by having the check for the file specified by SLURM_CONF go after the check for the default_slurm_config_file rather than before. Given that (at least to my understanding) configless operation is when that shouldn't exist (or if it does it'll be a symlink to the downloaded config) then that should be right order of operation anyway? What do you think? All the best, Chris Chris, The logic is a little bit different - SLURM_CONF had always precedence over built-in default it wasn't introduced as config-less, but it's rather in-use by config-less option. The new part is that SLURM_CONF gets set by salloc(and other commands) not its precedence over default location. If we ever decide to change this, such default can only be changed as part of a major release. When you think about that from a perspective of a random site using Slurm there are those who use environment modules just updating SLURM_CONF to achieve similar behavior. Another (maybe strongest) argument against that change is a "common sense" of built-in default configuration file, which seems to me being - use that location if nothing provided by other mechanisms like run time option or environment variable. Another way to consider, but definitively being a bigger change in your environment is Slurm multi-cluster setup, did you consider this option? cheers, Marcin Hi Marcin, Thanks for that - I can understand the reasoning behind that. Is it possible that SLURM_CONF could only be set when running configless (i.e. when "enable_configless" is set in SlurmctldParameters) ? That would seem like a reasonable compromise, this new behaviour would only occur when requested, leaving existing configurations unaffected by this change in behaviour. All the best, Chris Hi Marcin, Having poked around the code some more, and tried some patches myself, I've realised the simplest solution is probably to just use a task prolog to unset SLURM_CONF on our Cray XC systems. csamuel@gert01:~> cat /etc/slurm/taskprolog.sh #!/bin/bash # Unset SLURM_CONF on XC to prevent it breaking "module load esslurm" echo unset SLURM_CONF results in: csamuel@gert01:~> srun -q interactive -C haswell bash -c 'env | fgrep -c SLURM_CONF' srun: job 2224234 queued and waiting for resources srun: job 2224234 has been allocated resources 0 srun: error: nid00058: task 0: Exited with exit code 1 srun: Terminating job step 2224234.0 which is what we want. Likely the simplest solution all round (no patches, no wrappers, no changes to RPMs and only deployed where we need it via our ansible setup). All the best, Chris Chris, We had a short internal discussion on that case and we think that unset/override of SLURM_CONF in environment module for esslurm is the best approach. Did you check if the absolute path case is used within a job allocation (I asked Doug about that in comment 5) if it's not then there won't be any change for those? cheers, Marcin Chris, Just a touch base - are you OK with our advice? cheers, Marcin Chris, Another follow-up. If you don't reply within a week I'll close the report as information given, cheers, Marcin Chris, I'm closing the case as "information given". Should you have any question please don't hesitate to reopen. cheers, Marcin Hi Marcin, Sorry, I didn't see your previous replies until this was closed. We can't assume people will load the environment module, so we'll stick with unsetting it in the task prolog, that seems the easiest way to stop it breaking things for us. All the best, Chris Chris, Understood, however, you have to be aware that we're not running any regression testing with such a disruptive TaskProlog. This may result in some issues on your side in the future that may not be easy to reproduce by us without knowing about the variable being unset. cheers, Marcin (In reply to Marcin Stolarek from comment #18) > Understood, however, you have to be aware that we're not running any > regression testing with such a disruptive TaskProlog. This may result in > some issues on your side in the future that may not be easy to reproduce by > us without knowing about the variable being unset. Understood, it's just an unfortunate and unavoidable consequence of the Cray architecture. :-( |