Ticket 2007

Dear SchedMD,

When I pass a joblist to sacct with a hanging comma, slurmdbd crashes, e.g.,

$ sacct -X -j 132423,

and we get this in slurmdbd.log within a few minutes:

slurmdb_defs.c:385: : malloc failed

This is with slurmdbd 14.11.8 and sacct from Slurm 14.03.11. When the sacct is run from Slurm 14.11.9 then the behavior is odd but not problematic (see far below).

We were producing this when we nested squeue within sacct, with tr to convert the job list into a comma-separated list:

$ sacct -X -j `squeue -u myuser -tPD -h -r -o %A | tr '/\n/' ','`

If I were to remove that hanging comma, e.g., .. | sed 's/,$//', then the problem goes away.

While no one would deliberately set out to add a hanging comma like that, it could mean that slurmdbd is keeping a connection held up indefinitely. 

The strace shows three threads with a bunch of poll() timeouts which then becomes a bunch of recv() with EAGAIN errors, another one the same and then a bunch of mmap2() calls that meet ENOMEM before getting aborted. 

It is also noted that after such an sacct command is given on Slurm 14.03, even after Ctrl-C to break out of it, slurmdbd will be in a state where, within 10 minutes, memory utilization will start climbing up quickly past 1 GB, perhaps in reaction to other slurmdbd queries coming from scripts and users on our systems, but climb it will, and 3 GB within 25 minutes or so, which is enough on our 4 GB host for slurmdbd to fail malloc and die.

Slurm 14.11.9: odd but not problematic:

If sacct as above is given from Slurm 14.11.9, sacct appears to resolve the hanging comma by running the query successfully. However, it is not clear how it resolves it. If I feed it a nonexisting jobid with a hanging comma, it appears to fetch all jobs in the database. Likewise if I feed no jobid but provide a hanging comma, e.g.,

$ sacct -j ,

Not the same behavior as running scct without parameters, presumably because no jobs have run since the most recent midnight.

Just to summarize: SlurmDBD is running 14.11.8, the crash is induced by sacct with the hanging comma in the joblist from Slurm 14.03.11, no crash but odd results via sacct from Slurm 14.11.9.

Regards
Jeff