Ticket 4538

Created attachment 5773 [details]
slurmstld log file

Created attachment 5774 [details]
sacct output file

Hey Kolbeinn,

I'm able to reproduce the situation that you are are seeing.

Basically it's happening from a restart and having the large job ids still in the system when restarting. When restarting, Slurm sets it's next job id to give out to the highest jobid in the system + 1. So on your restart it started at the top again and then rerolled again.

I'm looking into a possible solution to prevent it from happening and will let you know what we find.

Just as a side note, jobs in the database are unique even though the jobids may roll. They can be distinguished by the submission time. By default sacct will only show the most recent job but the duplicates can be displayed using the --duplicates/-D option.

Thanks,
Brian

e.g.
brian@lappy:~/slurm/17.02/lappy$ scontrol show config | grep MaxJobId
MaxJobId                = 67043328

brian@lappy:~/slurm/17.02/lappy$ squeue
             JOBID PARTITION     NAME     USER ST       TIME   CPUS NODELIST(REASON)
brian@lappy:~/slurm/17.02/lappy$ sbatch -n1 --wrap="sleep 9999999999999"
Submitted batch job 67043321
brian@lappy:~/slurm/17.02/lappy$ sbatch -n1 --wrap="sleep 9999999999999"
Submitted batch job 67043322
brian@lappy:~/slurm/17.02/lappy$ sbatch -n1 --wrap="sleep 9999999999999"
Submitted batch job 67043323
brian@lappy:~/slurm/17.02/lappy$ sbatch -n1 --wrap="sleep 9999999999999"
Submitted batch job 67043324
brian@lappy:~/slurm/17.02/lappy$ sbatch -n1 --wrap="sleep 9999999999999"
Submitted batch job 67043325
brian@lappy:~/slurm/17.02/lappy$ sbatch -n1 --wrap="sleep 9999999999999"
Submitted batch job 67043326
brian@lappy:~/slurm/17.02/lappy$ sbatch -n1 --wrap="sleep 9999999999999"
Submitted batch job 67043327
brian@lappy:~/slurm/17.02/lappy$ sbatch -n1 --wrap="sleep 9999999999999"
Submitted batch job 1
brian@lappy:~/slurm/17.02/lappy$ sbatch -n1 --wrap="sleep 9999999999999"
Submitted batch job 2
brian@lappy:~/slurm/17.02/lappy$ sbatch -n1 --wrap="sleep 9999999999999"
Submitted batch job 3
brian@lappy:~/slurm/17.02/lappy$ sbatch -n1 --wrap="sleep 9999999999999"
Submitted batch job 4
brian@lappy:~/slurm/17.02/lappy$ sbatch -n1 --wrap="sleep 9999999999999"
Submitted batch job 5
brian@lappy:~/slurm/17.02/lappy$ sbatch -n1 --wrap="sleep 9999999999999"
Submitted batch job 6

brian@lappy:~/slurm/17.02/lappy$ squeue -Si
             JOBID PARTITION     NAME     USER ST       TIME   CPUS NODELIST(REASON)
                 1     debug     wrap    brian  R       0:05      2 lappy2
                 2     debug     wrap    brian  R       0:05      2 lappy2
                 3     debug     wrap    brian  R       0:05      2 lappy3
                 4     debug     wrap    brian  R       0:05      2 lappy3
                 5     debug     wrap    brian  R       0:05      2 lappy3
                 6     debug     wrap    brian  R       0:02      2 lappy4
          67043321     debug     wrap    brian  R       0:11      2 lappy1
          67043322     debug     wrap    brian  R       0:08      2 lappy1
          67043323     debug     wrap    brian  R       0:08      2 lappy1
          67043324     debug     wrap    brian  R       0:08      2 lappy1
          67043325     debug     wrap    brian  R       0:08      2 lappy2
          67043326     debug     wrap    brian  R       0:08      2 lappy2
          67043327     debug     wrap    brian  R       0:05      2 lappy3
brian@lappy:~/slurm/17.02/lappy$ for i in `seq 2 7`; do scancel 6704332$i; done;
brian@lappy:~/slurm/17.02/lappy$ for i in `seq 1 3`; do scancel $i; done;
brian@lappy:~/slurm/17.02/lappy$ squeue -Si
             JOBID PARTITION     NAME     USER ST       TIME   CPUS NODELIST(REASON)
                 4     debug     wrap    brian  R       0:22      2 lappy3
                 5     debug     wrap    brian  R       0:22      2 lappy3
                 6     debug     wrap    brian  R       0:19      2 lappy4
          67043321     debug     wrap    brian  R       0:28      2 lappy1


brian@lappy:~/slurm/17.02/lappy$ scontrol show jobs | grep JobId
JobId=67043321 JobName=wrap
JobId=4 JobName=wrap
JobId=5 JobName=wrap
JobId=6 JobName=wrap

brian@lappy:~/slurm/17.02/lappy$ echo restart
restart

brian@lappy:~/slurm/17.02/lappy$ sbatch -n1 --wrap="sleep 9999999999999"
Submitted batch job 67043323
brian@lappy:~/slurm/17.02/lappy$ sbatch -n1 --wrap="sleep 9999999999999"
Submitted batch job 67043324
brian@lappy:~/slurm/17.02/lappy$ sbatch -n1 --wrap="sleep 9999999999999"
Submitted batch job 67043325
brian@lappy:~/slurm/17.02/lappy$ sbatch -n1 --wrap="sleep 9999999999999"
Submitted batch job 67043326
brian@lappy:~/slurm/17.02/lappy$ sbatch -n1 --wrap="sleep 9999999999999"
Submitted batch job 67043327
brian@lappy:~/slurm/17.02/lappy$ sbatch -n1 --wrap="sleep 9999999999999"
Submitted batch job 1
brian@lappy:~/slurm/17.02/lappy$ sbatch -n1 --wrap="sleep 9999999999999"
Submitted batch job 2
brian@lappy:~/slurm/17.02/lappy$ sbatch -n1 --wrap="sleep 9999999999999"
Submitted batch job 3
brian@lappy:~/slurm/17.02/lappy$ sbatch -n1 --wrap="sleep 9999999999999"
Submitted batch job 7
brian@lappy:~/slurm/17.02/lappy$ sbatch -n1 --wrap="sleep 9999999999999"
Submitted batch job 8

brian@lappy:~/slurm/17.02/lappy$ squeue -Si
             JOBID PARTITION     NAME     USER ST       TIME   CPUS NODELIST(REASON)
                 1     debug     wrap    brian  R       0:04      2 lappy1
                 2     debug     wrap    brian  R       0:04      2 lappy2
                 3     debug     wrap    brian  R       0:04      2 lappy2
                 4     debug     wrap    brian  R       1:25      2 lappy3
                 5     debug     wrap    brian  R       1:25      2 lappy3
                 6     debug     wrap    brian  R       1:22      2 lappy4
                 7     debug     wrap    brian  R       0:04      2 lappy2
                 8     debug     wrap    brian  R       0:04      2 lappy2
          67043321     debug     wrap    brian  R       1:31      2 lappy1
          67043323     debug     wrap    brian  R       0:10      2 lappy3
          67043324     debug     wrap    brian  R       0:07      2 lappy3
          67043325     debug     wrap    brian  R       0:07      2 lappy1
          67043326     debug     wrap    brian  R       0:07      2 lappy1
          67043327     debug     wrap    brian  R       0:04      2 lappy4

The following patch has been added 17.11.1:
https://github.com/SchedMD/slurm/commit/7d83f77d64fcdf4ea2fb0670ffb8fdbba7f461a6

The patch will not set the next jobid to be the highest jobid in the system and will continue where it left before being restarted.

If you would like, you can patch this into 17.02.

Let us know if you have any questions.

Thanks,
Brian

Hi Brian,

Many thanks for your quick response and findings. As you have confirmed this bug does only affect us when we restart the slurmctld, then we can stay calm :)

We are in discussion if we will patch the current version, upgrade to 17.11.1 or possible just cancel all jobs with the high Id's and upgrade later.

Cheers,
Kolbeinn

FYI

We used the following workaround:
Set FirstJobId higher than last submitted job
Set MaxJobId lower than the old jobs still in queue
Restarted slurmctld service

In our case:
FirstJobId=1000001
MaxJobId=60000000

Results:
[2017-12-19T14:15:41.443] _slurm_rpc_submit_batch_job JobId=327382 usec=4635
[2017-12-19T14:15:41.444] _slurm_rpc_submit_batch_job JobId=327383 usec=193
[2017-12-19T14:15:42.018] _slurm_rpc_submit_batch_job JobId=327384 usec=164
[2017-12-19T14:15:42.509] Terminate signal (SIGINT or SIGTERM) received
### SLURMCTLD RESTART ###
[2017-12-19T14:15:55.340] slurmctld version 17.02.7 started on cluster lhpc
[2017-12-19T14:15:57.776] _slurm_rpc_submit_batch_job JobId=1000001 usec=299
[2017-12-19T14:15:58.788] _slurm_rpc_submit_batch_job JobId=1000002 usec=271
[2017-12-19T14:15:59.364] _slurm_rpc_submit_batch_job JobId=1000003 usec=783
[2017-12-19T14:16:00.266] _slurm_rpc_submit_batch_job JobId=1000004 usec=271

We will upgrade to patched version at our earliest convenience and change the JobId config back to normal.

Thanks for sharing. Good idea.

 Ég verð í fríi til föstudagsins 11. maí. Vinsamlegast sendið póst á helpdesk@decode.is ef erindið er áríðandi.

I'm on vacation until Mai 11. Please contact helpdesk@decode.is if urgent.

*** Ticket 5116 has been marked as a duplicate of this ticket. ***