Hi, We've observed this on 21.08 and 22.05.5. Sacct is segfaulting on large json output (several ten thousand jobs) slurmdbd is running on another machine, connected to mariadb on the same host. Memory resource consumption is high on the other host, but OK, there are no OOM. $ sacct --version slurm 22.05.5 Stacktrace: (gdb) run -A cms --start 2022-01-01 --end 2022-07-01 --json Starting program: /usr/bin/sacct -A cms --start 2022-01-01 --end 2022-07-01 --json [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Program received signal SIGSEGV, Segmentation fault. 0x00007ffff64af874 in printbuf_memappend () from /lib64/libjson-c.so.2 Missing separate debuginfos, use: debuginfo-install slurm-22.05.5-1.el7.x86_64 (gdb) (gdb) (gdb) (gdb) bt #0 0x00007ffff64af874 in printbuf_memappend () from /lib64/libjson-c.so.2 #1 0x00007ffff64afa56 in sprintbuf () from /lib64/libjson-c.so.2 #2 0x00007ffff64abebd in json_object_object_to_json_string () from /lib64/libjson-c.so.2 #3 0x00007ffff64abebd in json_object_object_to_json_string () from /lib64/libjson-c.so.2 #4 0x00007ffff64ac70a in json_object_array_to_json_string () from /lib64/libjson-c.so.2 #5 0x00007ffff64abebd in json_object_object_to_json_string () from /lib64/libjson-c.so.2 #6 0x00007ffff64ac70a in json_object_array_to_json_string () from /lib64/libjson-c.so.2 #7 0x00007ffff64abebd in json_object_object_to_json_string () from /lib64/libjson-c.so.2 #8 0x00007ffff64abd5c in json_object_to_json_string_ext () from /lib64/libjson-c.so.2 #9 0x00007ffff66b4b5a in serializer_p_serialize (dest=0x7fffffffc390, data=<optimized out>, flags=DATA_SER_FLAGS_PRETTY) at serializer_json.c:255 #10 0x00007ffff7a76a2e in data_g_serialize (dest=dest@entry=0x7fffffffc390, src=src@entry=0x6332e0, mime_type=<optimized out>, flags=flags@entry=DATA_SER_FLAGS_PRETTY) at data.c:2196 #11 0x00000000004047fb in dump_data (argc=argc@entry=8, argv=argv@entry=0x7fffffffc8e8) at data.c:238 #12 0x000000000040a48c in main (argc=8, argv=0x7fffffffc8e8) at sacct.c:201 Best, Erich
Thank you for the back trace. We will look into this and let you know if we need any more information.
Please provide the version of libjson-c installed and the output of the following: > ldd /lib64/libjson-c.so.2 > file /lib64/libjson-c.so.2 > objdump -s 3 /lib64/libjson-c.so.2
Created attachment 27550 [details] libjson objdump
Hi Nate, I'm not too familiar with objdump, but the -s3 option gives an error (objdump: '3': No such file), still the full dump is attached. # ldd /lib64/libjson-c.so.2 linux-vdso.so.1 => (0x00007ffd105ee000) libc.so.6 => /lib64/libc.so.6 (0x00007f2039992000) /lib64/ld-linux-x86-64.so.2 (0x00007f2039f6b000) # file /lib64/libjson-c.so.2 /lib64/libjson-c.so.2: symbolic link to `libjson-c.so.2.0.1' Sorry. I missed the info its running on Centos7. yum info is: Installed Packages Name : json-c Arch : x86_64 Version : 0.11 Release : 4.el7_0 Size : 64 k Repo : installed Summary : A JSON implementation in C URL : https://github.com/json-c/json-c/wiki License : MIT Description : JSON-C implements a reference counting object model that allows you to easily : construct JSON objects in C, output them as JSON formatted strings and parse : JSON formatted strings back into the C representation of JSON objects.
This looks like a duplicate of libjson-c issue: > https://github.com/json-c/json-c/pull/749 Please upgrade to at least json-c-0.16-20220414 release. Please tell me if help is needed in upgrading and re-compiling Slurm to link to the new libjson-c version.
Any updates?
Looks like there are no more questions. Closing out ticket.
