Ticket 14084

Created attachment 25062 [details]
slurm.conf

Here's a clean reproduction of it:

--

cseraphine@chhq-supgcmp001-14:40:44-/home/cseraphine$ srun -n 4  -w $HOSTNAME stress --vm 4  --vm-stride 8 --vm-hang 30 --vm-keep --vm-bytes 1G -t 200 &
[1] 32190
cseraphine@chhq-supgcmp001-14:40:56-/home/cseraphine$ stress: info: [32207] dispatching hogs: 0 cpu, 0 io, 4 vm, 0 hdd
stress: info: [32209] dispatching hogs: 0 cpu, 0 io, 4 vm, 0 hdd
stress: info: [32208] dispatching hogs: 0 cpu, 0 io, 4 vm, 0 hdd
stress: info: [32210] dispatching hogs: 0 cpu, 0 io, 4 vm, 0 hdd

cseraphine@chhq-supgcmp001-14:40:57-/home/cseraphine$ ps -u cseraphine -o pid,cmd,rss
    PID CMD                           RSS
  27087 /lib/systemd/systemd --user  9812
  27197 -bash                        8456
  32190 srun -n 4 -w chhq-supgcmp00  8352
  32193 srun -n 4 -w chhq-supgcmp00   888
  32207 /usr/bin/stress --vm 4 --vm   972
  32208 /usr/bin/stress --vm 4 --vm   968
  32209 /usr/bin/stress --vm 4 --vm   972
  32210 /usr/bin/stress --vm 4 --vm  1032
  32219 /usr/bin/stress --vm 4 --vm 1048948
  32220 /usr/bin/stress --vm 4 --vm 1048948
  32221 /usr/bin/stress --vm 4 --vm 1048948
  32222 /usr/bin/stress --vm 4 --vm 1048948
  32223 /usr/bin/stress --vm 4 --vm 1048948
  32224 /usr/bin/stress --vm 4 --vm 1048948
  32225 /usr/bin/stress --vm 4 --vm 1048948
  32226 /usr/bin/stress --vm 4 --vm 1048884
  32227 /usr/bin/stress --vm 4 --vm 1048948
  32228 /usr/bin/stress --vm 4 --vm 1048884
  32229 /usr/bin/stress --vm 4 --vm 1049004
  32230 /usr/bin/stress --vm 4 --vm 1048884
  32231 /usr/bin/stress --vm 4 --vm 1049004
  32232 /usr/bin/stress --vm 4 --vm 1048884
  32233 /usr/bin/stress --vm 4 --vm 1049004
  32234 /usr/bin/stress --vm 4 --vm 1049004
  32236 ps -u cseraphine -o pid,cmd  3568
cseraphine@chhq-supgcmp001-14:41:03-/home/cseraphine$ squeue -u cseraphine
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
           1475020 slurm21pm   stress cseraphi  R       0:38      1 chhq-supgcmp001
cseraphine@chhq-supgcmp001-14:41:34-/home/cseraphine$ sacct -j 1475020 --format jobid,jobname,state,maxrss
JobID           JobName      State     MaxRSS
------------ ---------- ---------- ----------
1475020          stress    RUNNING
cseraphine@chhq-supgcmp001-14:42:10-/home/cseraphine$ scontrol show job 1475020
JobId=1475020 JobName=stress
   UserId=cseraphine(29599) GroupId=cseraphine(29599) MCS_label=N/A
   Priority=3292 Nice=0 Account=gridadmins QOS=normal
   JobState=RUNNING Reason=None Dependency=(null)
   Requeue=1 Restarts=0 BatchFlag=0 Reboot=0 ExitCode=0:0
   RunTime=00:01:37 TimeLimit=01:00:00 TimeMin=N/A
   SubmitTime=2022-05-17T14:40:56 EligibleTime=2022-05-17T14:40:56
   AccrueTime=Unknown
   StartTime=2022-05-17T14:40:56 EndTime=2022-05-17T15:40:56 Deadline=N/A
   SuspendTime=None SecsPreSuspend=0 LastSchedEval=2022-05-17T14:40:56 Scheduler=Main
   Partition=slurm21pmain AllocNode:Sid=chhq-supgcmp001:27197
   ReqNodeList=chhq-supgcmp001 ExcNodeList=(null)
   NodeList=chhq-supgcmp001
   BatchHost=chhq-supgcmp001
   NumNodes=1 NumCPUs=4 NumTasks=4 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
   TRES=cpu=4,mem=4G,node=1,billing=4
   Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
   MinCPUsNode=1 MinMemoryCPU=1G MinTmpDiskNode=0
   Features=(null) DelayBoot=00:00:00
   OverSubscribe=YES Contiguous=0 Licenses=(null) Network=(null)
   Command=stress
   WorkDir=/home/cseraphine
   Power=


cseraphine@chhq-supgcmp001-14:42:33-/home/cseraphine$ fg
srun -n 4 -w $HOSTNAME stress --vm 4 --vm-stride 8 --vm-hang 30 --vm-keep --vm-bytes 1G -t 200
stress: info: [32208] successful run completed in 200s
stress: info: [32210] successful run completed in 200s
stress: info: [32207] successful run completed in 200s
stress: info: [32209] successful run completed in 200s
cseraphine@chhq-supgcmp001-14:44:16-/home/cseraphine$ sacct -j 1475020 --format jobid,jobname,state,maxrss
JobID           JobName      State     MaxRSS
------------ ---------- ---------- ----------
1475020          stress  COMPLETED      1036K

Hi,

After taking a look at your slurm.conf, it looks like you have the following configuration:

JobAcctGatherFrequency=energy=0,filesystem=0,network=0,task=0

If this values are set to 0, as you have, the job gathering is disabled until job termination, when it collects a unique metric (that is the not accurate "MaxRSS" value you are observing). The idea is to avoid any minimal interference of Slurm with the running processes in the node.

For the information you want, it is enough with enabling the task gathering, so, could you please repeat your test case changing the values (in slurm.conf) to:

JobAcctGatherFrequency=energy=0,filesystem=0,network=0,task=30

After the change, you will need to make it effective with: 

$ scontrol reconfigure

I am suggesting 30s for the frequency because it is the value we configure by default, but feel free to change it to a value that better fits your necessities. You will find more information here:

https://slurm.schedmd.com/slurm.conf.html#OPT_JobAcctGatherFrequency

Please, try the suggested changes, and let us know if they work as expected.

Kind regards,

Oscar

Oh good, I was hoping it was something stupid/easy like that, thank you!

The docs seemed to suggest that setting the intervals to zero would result in no data being gathered during the job's run, 

> If the task sampling interval is 0, accounting information is collected only at job termination (reducing Slurm interference with the job).

That is 100% what we are seeing, but I believe it should be made explicit that the MaxRSS information that _does_ get applied at job end might be garbage.  In *hindsight* this makes perfect sense (how do you see the peaks if you aren't looking at the process. during its lifecycle?) but apparently it is subtle enough to fool at least some Slurm admins :-(

Could we get few words tacked onto that sentence to reflect that?  Something like:

"(reducing Slurm interference with the job, and making accurate tracking of values such as MaxRSS impractical)"

Thanks!

Glad it worked out!
 
I see your point, the simple fact of having it accounted might be a little misleading. We will for sure take your comments into consideration. 

I am closing this bug for now. If you have any followup question, feel free to re-open it.

Regards,
Oscar