Ticket 1324

On SLURM 14.11, when scheduling a job with a "afterok" dependency depending on a job array, the job depending on the jobarray sometimes starts BEFORE all the jobs in the jobarray ran to completion (with an exit code of 0).

This is not happening every time but happens quite often. You can try to replicate it this way:

#!/bin/bash

# schedule a jobarray with two commands for every job:
## sleep a random number of seconds between 1 and 10
## print the time
jobid=$(sbatch --parsable --array=0-300 --wrap "sleep \$(( ( RANDOM % 10 ) + 1 )); echo \"ending job \$SLURM_ARRAY_TASK_ID at \$(date +%s.%N)\"")

# start the job depending on the jobarray and just print the date
sbatch --dependency="afterok:$jobid" --wrap "echo starting afterok dependency job at \$(date +%s.%N)"

If you look at the file contents and/or the slurm-*.out modification date, you will see that sometimes the date written in the outfile of the job depending on the jobarray is not the most recent and that some of the jobs in the jobarray actually finished AFTER the "afterok" job.

It also seems to me that this is happening only when the number of jobs in the jobarray is more than the maximum number of cores available, so jobs must wait for resources before starting.

This was not happening on previous SLURM releases (SLURM 14.03.4 was working perfectly fine).