Ticket 5416

Created attachment 7290 [details]
log snippet round job 66222894

Created attachment 7291 [details]
controller log snippet around job 66222894

Created attachment 7293 [details]
getgrouplist c prog

Hey Michael,

I believe I have an idea of where and what's happening. I'm able to reproduce the situation by placing a sleep() in the launch batch job code in the slurmd.

What I think is happening:
1. slurmctld allocates nodes for job
2. slurmctld sends launch batch job request to the slurmd
3. slurmd receives the message
4. the slurmd attempts to get the user's groups.

e.g.
	/* lookup gids if they weren't sent by slurmctld */
	if (!req->ngids)
		req->ngids = group_cache_lookup(req->uid, req->gid,
						req->user_name, &req->gids);

5. group_cache_loopup() -> _group_cache_lookup_internal()
6. _group_cache_lookup_internal() will get the users groups if it hasn't seen the user before OR if the entries are older than GroupUpdateTime (default:600 seconds).

	/* Cache lookup failed or entry value was too old, fetch new
	 * value and insert it into cache.  */
	while (getgrouplist(entry->username, entry->gid,
			    entry->gids, &entry->ngids) == -1) {

7. I'm guessing that getgrouplist is taking a long time (greater than MessageTimeout).
8. The slurmd tries to respond to the slurmctld who is waiting for a response on how the launch of the batch job went. But the slurmctld's launch request timed out so it's not there. So you get:

error: Could not confirm batch launch for job 66222894, aborting request

and then attempts to clean up and tells the slurmctld that the job failed. First attempting to the requeue the job -- which the slurmctld replies that the job couldn't be requeued because the job is not requeueable.

_slurm_rpc_requeue: 66222894: Requested operation is presently disabled

9. But the job is still requeued.

Requeuing JobID=66222876_18(66222894) State=0x0 NodeCnt=0

We need to look into why this is happening.

If the user were to submit another job to the node in the next 10 minutes, the jobs should run fine because the slurmd will use the cached entry, but if another job comes in after the 10 minutes the same situation will repeat.


So..... to help diagnose this and see if this is really the issue, can you run the attached c program (the test example from getgrouplist man page)?

e.g.
brian@lappy:~/ctests/getgrp$ time ./a.out brian 0
getgrouplist() returned -1; ngroups = 9

real    0m0.003s
user    0m0.000s
sys     0m0.003s

brian@lappy:~/ctests/getgrp$ time ./a.out brian 9
ngroups = 9
1003 (brian)
...

real    0m0.003s
user    0m0.000s
sys     0m0.003s



Let us know if this helps.

Thanks,
Brian

After you confirm the LDAP issues, you may want to look at using the LaunchParameters=send_gids. This might solve your issue now, but you might want to confirm any LDAP issues first.

https://slurm.schedmd.com/slurm.conf.html#OPT_send_gids

Hi

Thanks for the code- it did point out a serious issue with LDAP server performance that was masked by nscd (to get the bad numbers I would drop nscd caches with a restart of the service).  Anyway, enumerating group entries could take upwards of 100 seconds.

We're reverting to the old servers which should return us to our previous (hopefully) good state.  This change is going through our systems now so I should be able to all-clear soon.  We'll also be turning on `send_gids` as I think that will really help with some of our users' jobs (we have a lot of folks who submit thousands of jobs at a go)

I take it you're thinking the requeueing behavior might be a bug with slurm?  When we get group enumeration fixed I think we'll be out of the woods and wouldn't _need_ that fixed, but it'd be nice to get that working the way we'd expect.

Thanks for the help!

Michael

Good to hear! Glad to help.

For the requeue issue, Slurm is intentionally requeueing the job because it thinks that the job never launched so it's trying to reschedule the job. Slurm will only requeue the job like this in 3 situations -- 1. the batch job launch failed (this one), 2. the batch script is missing before launch and 3. the batch job credential failed to create.

Are you good to close the bug?

(In reply to Brian Christiansen from comment #6)
> Good to hear! Glad to help.
> 
> For the requeue issue, Slurm is intentionally requeueing the job because it
> thinks that the job never launched so it's trying to reschedule the job.
> Slurm will only requeue the job like this in 3 situations -- 1. the batch
> job launch failed (this one), 2. the batch script is missing before launch
> and 3. the batch job credential failed to create.

Ok- that makes sense.  Only thing was that it never did seem to re-run the job, though that may have been due to something else.  Maybe I'll try to replicate some other time.

> Are you good to close the bug?

Yep- I think so.  Thanks for the help.

Michael

In my situation the job runs after the requeue. Let me know if you see something different.

Thanks,
Brian