Ticket 853

Sorry, The Case Bug 853 is same contents of Bug 852.

Please close this Case (Bug 852).

The contents is duplicate.

*** Ticket 852 has been marked as a duplicate of this ticket. ***

(In reply to toru matsuoka from comment #0)
> Hello,Slurm Support team !
> 
> I'm Toru Matsuoka in Cray Japan Engineer.
> 
> Please　teach me about following contents.
> 
> We customer want use sstat commands. 
> 
> But , following error occured.
> 
> ■sstat command
> 
> Note: the sstat  command requires that the jobacct_gather plugin be
> installed and operational.
> 
> [root@mgmt2 slurm]# sstat --format=AveCPU,AvePages,AveRSS,AveVMSize,JobID -j
> 27005
>     AveCPU   AvePages     AveRSS  AveVMSize        JobID
> ---------- ---------- ---------- ---------- ------------
> sstat: error: Malformed RPC of type 5020 received
> sstat: error: slurm_receive_msgs: Header lengths are longer than data
> received
> sstat: error: Malformed RPC of type 5020 received
> sstat: error: slurm_receive_msgs: Header lengths are longer than data
> received
> sstat: error: Malformed RPC of type 5020 received
> sstat: error: slurm_receive_msgs: Header lengths are longer than data
> received
> sstat: error: slurm_job_step_stat: unknown return given from e035: 9001 rc =
> Communication connection failure
> sstat: error: slurm_job_step_stat: unknown return given from e036: 9001 rc =
> Communication connection failure
> sstat: error: slurm_job_step_stat: unknown return given from e034: 9001 rc =
> Communication connection failure
> sstat: error: problem getting step_layout for 27005.0: Communication
> connection failure

This is almost certainly due to differences in your slurm.conf file between nodes. Make sure that your slurm.conf file on the node where you execute sstat is identical to that of the compute nodes where job 27005 is running. RPC type 5020 is RESPONSE_JOB_STEP_STAT from the compute node.


> ■sacct command 
> 
> It look likes use sacct command.
> 
> [root@mgmt2 slurm]# sacct --j 27061
>        JobID    JobName  Partition    Account  AllocCPUS      State ExitCode
> ------------ ---------- ---------- ---------- ---------- ---------- --------
> 27061             prog4 ye016uta7+       root         40    RUNNING      0:0
> 27061.0       pmi_proxy                  root          2    RUNNING      0:0
> 
> 
> In Slurm.conf , 
> 
> ProctrackType=proctrack/pgid
> JobAcctGatherType parameter is not exist.
> 
> Is it necessary JobAcctGatherType in slurm.conf or cause of other problem?

Without a plugin to collect accounting information (e.g. memory, cpu use, etc.), that information will be reported as zero by sstat.

Hello,Slurm Support Team!

thanks for support.

Please teach me about following contents.

===================================================

Although our systems are CRAY CCS (CS300) products, the management node and the all computing node are using the same slurm.conf file fundamentally. 
(/opt/slurm/slurm/2.6.2/etc/sysconfig/slurm)
 
What kind of thing is it that such an error message outputs on such conditions? 

It was the same result although the sstat command was executed by the node by which the job is incidentally performed.


[root@e037 ~]# cd /opt/slurm/2.6.2/bin
[root@e037 bin]# ./sstat --format=AveCPU,AvePages,AveRSS,AveVMSize,JobID -j 27179
    AveCPU   AvePages     AveRSS  AveVMSize        JobID
---------- ---------- ---------- ---------- ------------
sstat: error: Malformed RPC of type 5020 received
sstat: error: slurm_receive_msgs: Header lengths are longer than data received
sstat: error: Malformed RPC of type 5020 received
sstat: error: slurm_receive_msgs: Header lengths are longer than data received
sstat: error: Malformed RPC of type 5020 received
sstat: error: slurm_receive_msgs: Header lengths are longer than data received
sstat: error: slurm_job_step_stat: unknown return given from e037: 9001 rc = Communication connection failure
sstat: error: slurm_job_step_stat: unknown return given from e038: 9001 rc = Communication connection failure
sstat: error: slurm_job_step_stat: unknown return given from e036: 9001 rc = Communication connection failure
sstat: error: problem getting step_layout for 27179.0: Communication connection failure


If JobAcctGatherType parameter is not set to a Slurm.conf file, are if all the information is displayed as 0, but If JobAcctGatherType parameter is set up in a slurm.conf file, isn't the sstat command displayed normally and become usable? 

===================================================
 
Best Regards..

Hello,Slurm Support team!

Thanks for your support ! 

I would like to check in a hurry about the following contents. 

Is JobAcctGatherType plug-in introduced together with introduction of SLURM? 
Or Is there any necessity of introducing independently? 
Or Is it OK only by describing in Slurm.conf? 

===============================================
・sstat can not running →

# squeue -l (part of slurm job)

　            27416 ye016uta7    prog4 uuta0026  RUNNING      20:15 3-00:00:00      2 e[042-043]

[root@mgmt2 ~]# ssh e042
Last login: Wed May 28 13:27:01 2014 from mgmt2

[root@e042 ~]# cd /opt/slurm/2.6.2./bin
[root@e042 bin]# ls
generate_pbs_nodefile  sacct     sinfo        sstat
mpiexec                sacctmgr  sjobexitmod  strigger
pbsnodes               salloc    sjstat       sview
qdel                   sattach   smap         weekqueue.sh
qhold                  sbatch    sprio        weekqueue.sh.20140203
qrls                   sbcast    squeue       weekqueue.sh.20140314
qstat                  scancel   sreport      weekqueue.sh.20140319
qsub                   scontrol  srun
queue_up_down.sh       sdiag     sshare

[root@e042 bin]# ./sstat
sstat: error: You didn't give me any jobs to stat.

[root@e042 bin]# ./sstat --j 27416
       JobID  MaxVMSize  MaxVMSizeNode  MaxVMSizeTask  AveVMSize     MaxRSS MaxRSSNode MaxRSSTask     AveRSS MaxPages MaxPagesNode   MaxPagesTask   AvePages     MinCPU MinCPUNode MinCPUTask     AveCPU   NTasks AveCPUFreq ReqCPUFreq ConsumedEnergy  MaxDiskRead MaxDiskReadNode MaxDiskReadTask  AveDiskRead MaxDiskWrite MaxDiskWriteNode MaxDiskWriteTask AveDiskWrite
------------ ---------- -------------- -------------- ---------- ---------- ---------- ---------- ---------- -------- ------------ -------------- ---------- ---------- ---------- ---------- ---------- -------- ---------- ---------- -------------- ------------ --------------- --------------- ------------ ------------ ---------------- ---------------- ------------
sstat: error: Malformed RPC of type 5020 received
sstat: error: slurm_receive_msgs: Header lengths are longer than data received
sstat: error: Malformed RPC of type 5020 received
sstat: error: slurm_receive_msgs: Header lengths are longer than data received
sstat: error: slurm_job_step_stat: unknown return given from e042: 9001 rc = Communication connection failure
sstat: error: slurm_job_step_stat: unknown return given from e043: 9001 rc = Communication connection failure
sstat: error: problem getting step_layout for 27416.0: Communication connection failure


Best Regards..
Toru Matsuoka

(In reply to toru matsuoka from comment #5)
> Hello,Slurm Support team!
> 
> Thanks for your support ! 
> 
> I would like to check in a hurry about the following contents. 
> 
> Is JobAcctGatherType plug-in introduced together with introduction of SLURM? 
> Or Is there any necessity of introducing independently? 
> Or Is it OK only by describing in Slurm.conf? 

I do not understand your questions.

You do need to configure JobAcctGatherType in slurm.conf.

Did you change slurm.conf and not restart the slurm daemons (so they would be using old configuration information)?

Thank you for your support.

I understood it.

>You do need to configure JobAcctGatherType in slurm.conf.
>Did you change slurm.conf and not restart the slurm daemons 
>(so they would be using old configuration information)?

We and customer done configure JobAcctGatherType in slurm.conf
and change slurm.conf and not restart the slurm daemons.

We and customer will scontrol reconfigure after configure 
JobAcctGatherType in slurm.conf.

Best Regards..
Toru Matsuoka

Please keep in mind a simple scontrol reconfig will not load the setting correctly.

You need to restart the daemons.

Restart the slurmctld as well as all the slurmd's after the configuration change.

Hello,Slurm Support team! 

I understood following contents and illustrated for customer. 

>Please keep in mind a simple scontrol reconfig will not load the setting >correctly.
>You need to restart the daemons.
>Restart the slurmctld as well as all the slurmd's after the configuration >change.

Thank you!

If add question is occured,I will open as new case.

Please close this case!

Best Regards..
Toru Matsuoka

Closed per customer request. SLurm daemons needed to be restarted to enable change in accounting information collection plugin.