Ticket 13614

Created attachment 23864 [details]
slurm.conf

Created attachment 23865 [details]
slurmdbd log

Hi Thomas,

I don't see correlation between slurmdbd.log and slurmctld.log. The last line of dbd log is older than:

[2022-03-13T11:45:02.545] error: unpackmem_xmalloc: Buffer to be unpacked is too large (1735288180 > 100000000)

Additionally, I only see the above error of this type in the slurmctld.log, and it doesn't seem to me like it's connected to any other log lines.

It's very very difficult then to give you an explanation for this. I know some scenarios where this could be triggered. All them are in situations where the ctld is unpacking an incoming RPC, and realizes it is just to big that it doesn't make any sense.

We need then to move our focus out of ctld, which is doing expected behaviour, and seek for the one who's sending that big message. A user trying to send a sbatch with a very big script is just one example.

It would be a good idea to have a look at slurmd.log as well. I know there's plenty of them, and we don't really know the nodes to look at, but a pdsh command looking for the number 1735288180 in logs would be a good starting point.

Finally, let's lower to sev-3 the issue. I don't think this is making a continuous impact in the cluster operations. If I'm wrong please, consider extending the explanation of the issue by providing information on how it's affecting the cluster.

Tell us if you find something else.

Regards,
Carlos.

I'll check teh slurmd.logs and report back

Hi Carlos, Sadly, while we did find a few OOM errors like this, they were in much older logs. Nothing to correspond to the death of slurmctld last week. Any other ideas?

> we did find a few OOM errors like this

I don't expect an OOM to cause this particular error. As I said, this error is not a controller's real error (it doesn't break anything). Controller is only telling that it has tried to unpack something that is just too big to be a correct RPC, so it just gives away thit RPC.

> Nothing to correspond to the death of slurmctld last week.
> Any other ideas?

I can imagine a user or script with a bug. Something like:

batch [big file]

I know it seems quite illogic, but think of some HPC application like AMBER, GROMACS, WRF, etc. They use sometimes quite big input files. If they're called like:

sbatch --wrap '[APP] [params]'

I can imagine by chance someone having a mistake and sbatch ending up treating a big input file as [APP].

Other option in sbatch while reading script from STDIN, scripted, and some unbound loop writing there.

If it's just 1 error, once in the past, don't pay much attention on this. Specially if you don't see any corresponding problem in any of the slurmd logs across the cluster.

Cheers,
Carlos.

Closing now as info give. Please reopen if needed.

Cheers,
Carlos.