I have been playing around with hidden partitions and noticed that a regular squeue can't see jobs when they are in both a hidden and visible partition... 20140514154413 lud1:stuartm> squeue -u stuartm JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 20140514154420 lud1:stuartm> scontrol show job 45750 JobId=45750 Name=test UserId=stuartm(3005) GroupId=prod(2000) Priority=100 Nice=0 Account=(null) QOS=normal JobState=PENDING Reason=Resources Dependency=(null) Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0 RunTime=00:00:00 TimeLimit=01:00:00 TimeMin=N/A SubmitTime=2014-05-14T15:44:16 EligibleTime=2014-05-14T15:44:16 StartTime=Unknown EndTime=Unknown PreemptTime=None SuspendTime=None SecsPreSuspend=0 Partition=teambondIdle,idle AllocNode:Sid=lud1:51538 ReqNodeList=(null) ExcNodeList=(null) NodeList=(null) NumNodes=1-1 NumCPUs=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* Socks/Node=* NtasksPerN:B:S:C=1:0:*:* CoreSpec=0 MinCPUsNode=1 MinMemoryNode=953M MinTmpDiskNode=0 Features=(null) Gres=(null) Reservation=(null) Shared=1 Contiguous=0 Licenses=(null) Network=(null) Command=/l1/000scratch/stuartm/rj.test.rYW6ji WorkDir=/l1/000scratch/stuartm StdErr=/l1/000scratch/stuartm/logs/test.o45750 StdIn=/dev/null StdOut=/l1/000scratch/stuartm/logs/test.o45750 20140514154426 lud1:stuartm> squeue -a -u stuartm JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 45750 teambondI test stuartm PD 0:00 1 (Resources) Partition Idle is hidden and partition teambondIdle is visble. If a job is only on a hidden partition, I can envisage that is might be hidden... but in both a hidden and visible? I don't quite know what the rationale for hidden partitions is.
This will be fixed in v14.03.4. https://github.com/SchedMD/slurm/commit/5fc21da299423bfda9e24e0936fc5df2de0d276a