Ticket 2349

FrontEnd is meant for system like IBM BlueGene or Cray where jobs need some special handling to launch on the system itself, you should not use that here.

Systems with a frontend always launch all tasks on one of the frontend nodes - thus you're seeing your frontend's hostname in your jobs. The resources on those requested nodes are being reserved (thus sinfo / squeue display that list) but your job would need to know to connect out from the frontend.

I'd recommend rebuilding with much fewer options, this should give you a fully functional build:

./configure --prefix=/opt/slurm

A lot of the options you'd enabled are for specialized hardware platforms or development, and don't apply to your config.

I'd also recommend downloading 15.08.6 - there are a few important fixes in there you'll want. 15.08.7 is due out within the next few weeks and will have some moderate-severity fixes as well.

For your config, please remove FrontendNode entirely, and I'd also suggest setting SelectType=select/cons_res and SelectTypeParameters=CR_Core_Mem if you want to allocate specific amounts of memory and cpu cores within nodes to each job. (select/linear will always allocate entire nodes at a time.)

If you're using slurmdbd (which we highly recommend) that will have a record of all jobs - JobCompType=jobcomp/filetxt will produce information that you'd already have using the `sacct` tool, so I'd suggest removing that as well.

- Tim

Hi Tim,

Thank you for your prompt response. I found a thread which said the same thing. Now, it's all clear that I will have to recompile. I am used to torque/Moab and had to use multiple configure options for what I wanted. Especially, pam option to make sure users don't get onto nodes if they don't have jobs running.

Right now, at the beginning of the testing it won't matter much but we will definitely need pam support.  From the configure options I couldn't find whether it is enabled by default.

Either way, I will do what you suggested.

Thank you again,
Best,
Sreedhar.

On 01/14/2016 04:24 PM, bugs@schedmd.com wrote:
> Right now, at the beginning of the testing it won't matter much but we will
> definitely need pam support.  From the configure options I couldn't find
> whether it is enabled by default.
>
> Either way, I will do what you suggested.
>

You'll need to install the pam_slurm.so module manually, the installer 
doesn't try to automatically load it into your pam stack.

See contribs/pam/README has notes on how to set it up. Please note that 
if you manually install the pam_slurm.so file into /usr/lib you'll want 
to make sure to update this on each major release.

There's a new feature called pam_slurm_adopt that was built by Ryan Cox 
at BYU which works with cgroups (you'll need TaskPlugin=cgroup setup and 
working as well, http://slurm.schedmd.com/cgroup.conf.html) to constrain 
incoming user SSH sessions into the appropriate cgroup. There are some 
notes as to how to set this up in contribs/pam_slurm_adopt.

Hi,

I have recompiled and I am seeing this error. Ports are open and they're able to talk. Is there anything else I can do here? 

Thanks,
Sreedhar.

root@caslurmdev1:/u/sreedhar/jobs$ sinfo -N -l
Thu Jan 14 18:41:01 2016
NODELIST     NODES   PARTITION       STATE CPUS    S:C:T MEMORY TMP_DISK WEIGHT FEATURES REASON
caslurmdev1      1 production*        idle    4    1:4:1   7816        0      1   (null) none
caslurmdev2      1 production*        down    4    1:4:1   7816        0      1   (null) Not responding


root@caslurmdev1:/u/sreedhar/jobs$ scontrol show node caslurmdev2
NodeName=caslurmdev2 Arch=x86_64 CoresPerSocket=4
   CPUAlloc=0 CPUErr=0 CPUTot=4 CPULoad=0.01 Features=(null)
   Gres=(null)
   NodeAddr=caslurmdev2 NodeHostName=caslurmdev2 Version=15.08
   OS=Linux RealMemory=7816 AllocMem=0 FreeMem=5692 Sockets=1 Boards=1
   State=DOWN ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A
   BootTime=2015-10-30T15:47:22 SlurmdStartTime=2016-01-14T18:34:17
   CapWatts=n/a
   CurrentWatts=0 LowestJoules=0 ConsumedJoules=0
   ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s
   Reason=Not responding [slurm@2016-01-14T17:20:50]


caslurmdev2:~/slurm-15.08.6# scontrol show slurmd
Active Steps             = NONE
Actual CPUs              = 4
Actual Boards            = 1
Actual sockets           = 1
Actual cores             = 4
Actual threads per core  = 1
Actual real memory       = 7816 MB
Actual temp disk space   = 116675 MB
Boot time                = 2016-01-14T18:34:17
Hostname                 = caslurmdev2
Last slurmctld msg time  = 2016-01-14T18:37:56
Slurmd PID               = 33008
Slurmd Debug             = 6
Slurmd Logfile           = /var/log/slurm/slurmd.log
Version                  = 15.08.6

Check slurm daemon log files 

On January 14, 2016 7:32:41 PM PST, bugs@schedmd.com wrote:
>http://bugs.schedmd.com/show_bug.cgi?id=2349
>
>--- Comment #4 from Sreedhar <sreedharnyu@gmail.com> ---
>Hi,
>
>I have recompiled and I am seeing this error. Ports are open and
>they're able
>to talk. Is there anything else I can do here? 
>
>Thanks,
>Sreedhar.
>
>root@caslurmdev1:/u/sreedhar/jobs$ sinfo -N -l
>Thu Jan 14 18:41:01 2016
>NODELIST     NODES   PARTITION       STATE CPUS    S:C:T MEMORY
>TMP_DISK WEIGHT
>FEATURES REASON
>caslurmdev1      1 production*        idle    4    1:4:1   7816       
>0      1
>  (null) none
>caslurmdev2      1 production*        down    4    1:4:1   7816       
>0      1
>  (null) Not responding
>
>
>root@caslurmdev1:/u/sreedhar/jobs$ scontrol show node caslurmdev2
>NodeName=caslurmdev2 Arch=x86_64 CoresPerSocket=4
>   CPUAlloc=0 CPUErr=0 CPUTot=4 CPULoad=0.01 Features=(null)
>   Gres=(null)
>   NodeAddr=caslurmdev2 NodeHostName=caslurmdev2 Version=15.08
>   OS=Linux RealMemory=7816 AllocMem=0 FreeMem=5692 Sockets=1 Boards=1
>   State=DOWN ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A
>   BootTime=2015-10-30T15:47:22 SlurmdStartTime=2016-01-14T18:34:17
>   CapWatts=n/a
>   CurrentWatts=0 LowestJoules=0 ConsumedJoules=0
>   ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s
>   Reason=Not responding [slurm@2016-01-14T17:20:50]
>
>
>caslurmdev2:~/slurm-15.08.6# scontrol show slurmd
>Active Steps             = NONE
>Actual CPUs              = 4
>Actual Boards            = 1
>Actual sockets           = 1
>Actual cores             = 4
>Actual threads per core  = 1
>Actual real memory       = 7816 MB
>Actual temp disk space   = 116675 MB
>Boot time                = 2016-01-14T18:34:17
>Hostname                 = caslurmdev2
>Last slurmctld msg time  = 2016-01-14T18:37:56
>Slurmd PID               = 33008
>Slurmd Debug             = 6
>Slurmd Logfile           = /var/log/slurm/slurmd.log
>Version                  = 15.08.6
>
>-- 
>You are receiving this mail because:
>You are watching all bug changes.

Please ignore my previous comment. Everything is working fine now.

Thanks,
Sreedhar.

(In reply to Sreedhar from comment #6)
> Please ignore my previous comment. Everything is working fine now.
> 
> Thanks,
> Sreedhar.