Ticket 15272

Summary:	sacct segfault on json output
Product:	Slurm	Reporter:	Erich Birngruber <erich.birngruber>
Component:	Accounting	Assignee:	Nate Rini <nate>
Status:	RESOLVED INFOGIVEN	QA Contact:
Severity:	4 - Minor Issue
Priority:	---
Version:	22.05.5
Hardware:	Linux
OS:	Linux
Site:	IMP	Alineos Sites:	---
Atos/Eviden Sites:	---	Confidential Site:	---
Coreweave sites:	---	Cray Sites:	---
DS9 clusters:	---	HPCnow Sites:	---
HPE Sites:	---	IBM Sites:	---
NOAA SIte:	---	OCF Sites:	---
Recursion Pharma Sites:	---	SFW Sites:	---
SNIC sites:	---	Linux Distro:	CentOS
Machine Name:		CLE Version:
Version Fixed:		Target Release:	---
DevPrio:	---	Emory-Cloud Sites:	---
Attachments:	libjson objdump

Description Erich Birngruber 2022-10-25 03:59:23 MDT

Hi,
We've observed this on 21.08 and 22.05.5.
Sacct is segfaulting on large json output (several ten thousand jobs)
slurmdbd is running on another machine, connected to mariadb on the same host.
Memory resource consumption is high on the other host, but OK, there are no OOM.

$ sacct --version
slurm 22.05.5

Stacktrace:
(gdb) run -A cms --start 2022-01-01 --end 2022-07-01 --json 
Starting program: /usr/bin/sacct -A cms --start 2022-01-01 --end 2022-07-01 --json
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff64af874 in printbuf_memappend () from /lib64/libjson-c.so.2
Missing separate debuginfos, use: debuginfo-install slurm-22.05.5-1.el7.x86_64
(gdb) 
(gdb) 
(gdb) 
(gdb) bt
#0  0x00007ffff64af874 in printbuf_memappend () from /lib64/libjson-c.so.2
#1  0x00007ffff64afa56 in sprintbuf () from /lib64/libjson-c.so.2
#2  0x00007ffff64abebd in json_object_object_to_json_string () from /lib64/libjson-c.so.2
#3  0x00007ffff64abebd in json_object_object_to_json_string () from /lib64/libjson-c.so.2
#4  0x00007ffff64ac70a in json_object_array_to_json_string () from /lib64/libjson-c.so.2
#5  0x00007ffff64abebd in json_object_object_to_json_string () from /lib64/libjson-c.so.2
#6  0x00007ffff64ac70a in json_object_array_to_json_string () from /lib64/libjson-c.so.2
#7  0x00007ffff64abebd in json_object_object_to_json_string () from /lib64/libjson-c.so.2
#8  0x00007ffff64abd5c in json_object_to_json_string_ext () from /lib64/libjson-c.so.2
#9  0x00007ffff66b4b5a in serializer_p_serialize (dest=0x7fffffffc390, data=<optimized out>, flags=DATA_SER_FLAGS_PRETTY) at serializer_json.c:255
#10 0x00007ffff7a76a2e in data_g_serialize (dest=dest@entry=0x7fffffffc390, src=src@entry=0x6332e0, mime_type=<optimized out>, flags=flags@entry=DATA_SER_FLAGS_PRETTY) at data.c:2196
#11 0x00000000004047fb in dump_data (argc=argc@entry=8, argv=argv@entry=0x7fffffffc8e8) at data.c:238
#12 0x000000000040a48c in main (argc=8, argv=0x7fffffffc8e8) at sacct.c:201


Best,
Erich

Comment 1 Jason Booth 2022-10-25 20:39:47 MDT

Thank you for the back trace. We will look into this and let you know if we need any more information.

Comment 3 Nate Rini 2022-10-28 21:23:39 MDT

Please provide the version of libjson-c installed and the output of the following:
> ldd /lib64/libjson-c.so.2
> file /lib64/libjson-c.so.2
> objdump -s 3 /lib64/libjson-c.so.2

Comment 4 Erich Birngruber 2022-11-02 02:22:50 MDT

Created attachment 27550 [details]
libjson objdump

Comment 5 Erich Birngruber 2022-11-02 02:25:10 MDT

Hi Nate,

I'm not too familiar with objdump, but the -s3 option gives an error (objdump: '3': No such file), still the full dump is attached.

# ldd /lib64/libjson-c.so.2
	linux-vdso.so.1 =>  (0x00007ffd105ee000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f2039992000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f2039f6b000)

# file /lib64/libjson-c.so.2
/lib64/libjson-c.so.2: symbolic link to `libjson-c.so.2.0.1'


Sorry. I missed the info its running on Centos7. yum info is:

Installed Packages
Name        : json-c
Arch        : x86_64
Version     : 0.11
Release     : 4.el7_0
Size        : 64 k
Repo        : installed
Summary     : A JSON implementation in C
URL         : https://github.com/json-c/json-c/wiki
License     : MIT
Description : JSON-C implements a reference counting object model that allows you to easily
            : construct JSON objects in C, output them as JSON formatted strings and parse
            : JSON formatted strings back into the C representation of JSON objects.

Comment 7 Nate Rini 2022-11-02 13:15:34 MDT

This looks like a duplicate of libjson-c issue:
> https://github.com/json-c/json-c/pull/749

Please upgrade to at least json-c-0.16-20220414 release.

Please tell me if help is needed in upgrading and re-compiling Slurm to link to the new libjson-c version.

Comment 8 Nate Rini 2022-11-15 09:56:46 MST

Any updates?

Comment 9 Nate Rini 2022-12-05 13:34:32 MST

Looks like there are no more questions. Closing out ticket.