Ticket 5645

Hi there,

We use GRES to control access to temp disk space (the submit filter sets it for the user based on their --tmp request) and it's worked nicely up until now.

However, we've just upgraded our 8 high memory nodes from 350GB temp to 2TB local temp, and the gres.conf now looks like:

# Temporary directory space
NodeName=john[1-107],gina[1-4] Name=tmp Count=350G
NodeName=bryan[1-8] Name=tmp Count=1900G

We can see that slurmd picks that up fine:

slurmd: Gres Name=tmp Type=(null) Count=2040109465600

and:

[root@bryan7 ~]# scontrol show slurmd
Active Steps             = NONE
Actual CPUs              = 36
Actual Boards            = 1
Actual sockets           = 2
Actual cores             = 18
Actual threads per core  = 1
Actual real memory       = 772475 MB
Actual temp disk space   = 1906795 MB
Boot time                = 2018-08-30T16:09:02
Hostname                 = bryan7
Last slurmctld msg time  = 2018-08-30T16:18:28
Slurmd PID               = 4336
Slurmd Debug             = 8
Slurmd Logfile           = (null)
Version                  = 17.11.7


but this does not seem to get reflected in the control daemon:

[root@bryan7 ~]# scontrol show node bryan7
NodeName=bryan7 Arch=x86_64 CoresPerSocket=18
   CPUAlloc=0 CPUErr=0 CPUTot=36 CPULoad=0.02
   AvailableFeatures=(null)
   ActiveFeatures=(null)
   Gres=gpu:p100:2,tmp:350G
   NodeAddr=bryan7 NodeHostName=bryan7 Version=17.11
   OS=Linux 3.10.0-862.9.1.el7.x86_64 #1 SMP Mon Jul 16 16:29:36 UTC 2018 
   RealMemory=768000 AllocMem=0 FreeMem=766687 Sockets=2 Boards=1
   State=DOWN+DRAIN ThreadsPerCore=1 TmpDisk=1900000 Weight=2000 Owner=N/A MCS_label=N/A
   Partitions=skylake,skylake-gpu,debug 
   BootTime=2018-08-30T15:36:15 SlurmdStartTime=2018-08-30T16:09:02
   CfgTRES=cpu=36,mem=750G,billing=36,gres/gpu=2,gres/tmp=375809638400
   AllocTRES=
   CapWatts=n/a
   CurrentWatts=270 LowestJoules=569132 ConsumedJoules=152867
   ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s
   Reason=SW_nvme [rhumble@2018-08-30T14:41:29]

You can see the CfgTRES line still has the old version, even after multiple slurmd restarts and a reboot of the compute node as well as a restart of slurmctld:

   CfgTRES=cpu=36,mem=750G,billing=36,gres/gpu=2,gres/tmp=375809638400

Looking at the web page we can see:

https://slurm.schedmd.com/gres.html

# By default a node has no generic resources and its maximum
# count is 4,294,967,295.

Now our count is 2,040,109,465,600 which is a lot larger.

However, the manual page says instead:

# By default a node has no generic resources and its maximum
# count is that of an unsigned 64bit integer.

which should be large enough for just under 2TB.

I'm not seeing any messages in our slurmctld logs that would be relevant to this.

All the best,
Chris