Ticket 7127 - Tons of "Unable to read requested pattern in /proc/148502/status" in logs
Summary: Tons of "Unable to read requested pattern in /proc/148502/status" in logs
Status: RESOLVED DUPLICATE of ticket 6501
Alias: None
Product: Slurm
Classification: Unclassified
Component: slurmd (show other tickets)
Version: 18.08.7
Hardware: Linux Linux
: 3 - Medium Impact
Assignee: Director of Support
QA Contact:
URL:
Depends on:
Blocks:
 
Reported: 2019-05-28 09:29 MDT by Anthony DelSorbo
Modified: 2019-05-28 10:19 MDT (History)
0 users

See Also:
Site: NOAA
Slinky Site: ---
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
Google sites: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: NESCC
NoveTech Sites: ---
Nvidia HWinf-CS Sites: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Tzag Elita Sites: ---
Linux Distro: ---
Machine Name: Any
CLE Version:
Version Fixed:
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this ticket.
Description Anthony DelSorbo 2019-05-28 09:29:14 MDT 
This is the first of three tickets concerning slurm's inability to "read" various file systems.  Here, I'm seeing:

slurmstepd[148497]: debug3: jobacct_gather_linux: unable to read requested pattern in /proc/148502/status

Here's a sample count of hits for just part of today 

grep -c "unable to read requested pattern" /log/syslogs/2019.05.28/t05*
/log/syslogs/2019.05.28/t0500:23061
/log/syslogs/2019.05.28/t0501:22165
/log/syslogs/2019.05.28/t0502:26200
/log/syslogs/2019.05.28/t0503:20169
/log/syslogs/2019.05.28/t0504:21024
/log/syslogs/2019.05.28/t0505:41840
/log/syslogs/2019.05.28/t0506:29901
/log/syslogs/2019.05.28/t0507:29695

Would you please guide me in understanding why the slurmd daemons are spewing out so many of these and how to correct the issue?  Short of turning off debug3, there must be some setting that will permit me to reduce or eliminate these messages from the logs.

Thanks,

Tony.
Comment 2 Jason Booth 2019-05-28 10:19:49 MDT 
Hi Tony,

This is resolved in the following commit which is part of the 19.05 release.

https://github.com/SchedMD/slurm/commit/585b81d0424526ce7906da8e0818476f1aa3ba4a

For now, I would suggest you lower your log level since these messages will be dumped at debug2 or higher and a production system should not go beyond "debug" unless absolutely needed.

*** This ticket has been marked as a duplicate of ticket 6501 ***