What does this error mean? Aug 11 14:27:19 holy-slurm02.rc.fas.harvard.edu slurmctld[1468883]: error: gres_select_filter_select_and_set job 38196442 failed to satisfy gres-per-job counter This is the job in question: [root@holy-slurm02 ~]# scontrol show job 38196442 JobId=38196442 ArrayJobId=38196442 ArrayTaskId=0 JobName=npg_i1p10_001s UserId=nabreu(64363) GroupId=barak_lab(5117) MCS_label=N/A Priority=4997332 Nice=0 Account=kempner_barak_lab QOS=normal JobState=PENDING Reason=Priority Dependency=(null) Requeue=1 Restarts=11 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:00:00 TimeLimit=2-00:00:00 TimeMin=N/A SubmitTime=2026-08-11T13:41:58 EligibleTime=2026-08-11T13:43:59 AccrueTime=2026-08-11T13:43:59 StartTime=2026-08-12T02:07:20 EndTime=2026-08-14T02:07:20 Deadline=N/A SuspendTime=None SecsPreSuspend=0 LastSchedEval=2026-08-11T14:29:07 Scheduler=Main Partition=kempner_requeue AllocNode:Sid=0.0.0.0:870555 ReqNodeList=(null) ExcNodeList=(null) NodeList= SchedNodeList=holygpu8a11101 StepMgrEnabled=Yes NumNodes=1-1 NumCPUs=32 NumTasks=1 CPUs/Task=32 ReqB:S:C:T=0:0:*:* ReqTRES=cpu=32,mem=400G,node=1,billing=1739,gres/gpu=4 AllocTRES=(null) Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=32 MinMemoryNode=400G MinTmpDiskNode=0 Features=h200|h100 DelayBoot=00:00:00 OverSubscribe=NO Exclusive=NO Contiguous=0 Licenses=(null) LicensesAlloc=(null) Network=(null) Command=/n/home07/nabreu/opt-rl/scripts/run_npg_grpo_ours_full_i1p10_qwen25math_dapo_aime_v5_b128_requeue_h200.sh SubmitLine=sbatch --array=0-0 --job-name=npg_i1p10_001s --export=ALL,CONDA_ENV_PREFIX=/n/netscratch/kempner_barak_lab/Lab/nabreu/.mamba/envs/opt-rl-scratch,SWEEP_GRID=../configs-na/grid_npg_grpo_ours_full_i1p10_qwen25math_1_5b_dapo_aime_v5_b128_kl0.001.yaml,SWEEP_EXTRA_OVERRIDES=ckpt.save_dir=/n/netscratch/kempner_barak_lab/Lab/nabreu/opt-rl/ckpts/npg-ours-full-i1p10-qwen25math-1.5b-dapo-aime-v5-b128-kl0.001-37693959-0 scripts/run_npg_grpo_ours_full_i1p10_qwen25math_dapo_aime_v5_b128_requeue_h200.sh WorkDir=/n/home07/nabreu/opt-rl StdErr=/n/home07/nabreu/opt-rl/printouts/npg_ours_full_i1p10/38196442_0_npg_ours_i1p10_v5b128.err StdIn=/dev/null StdOut=/n/home07/nabreu/opt-rl/printouts/npg_ours_full_i1p10/38196442_0_npg_ours_i1p10_v5b128.out TresPerJob=gres/gpu:4 TresPerTask=cpu=32 Here is the start of the job script: [root@holy-slurm02 ~]# sacct -B -j 38196442 Batch Script for 38196442_0 -------------------------------------------------------------------------------- #!/bin/bash #SBATCH -J npg_i1p10 #SBATCH -o printouts/npg_ours_full_i1p10/%A_%a_npg_ours_i1p10_v5b128.out #SBATCH -e printouts/npg_ours_full_i1p10/%A_%a_npg_ours_i1p10_v5b128.err #SBATCH -p kempner_requeue #SBATCH -N 1 #SBATCH --gpus=4 #SBATCH --cpus-per-task=32 #SBATCH --mem=400G #SBATCH -t 2-00:00:00 #SBATCH --array=0-2 #SBATCH --constraint="h200|h100" #SBATCH --account=kempner_barak_lab #SBATCH --requeue #SBATCH --signal=B:USR1@600 I don't see anything obviously wrong.
Hi, Can you share your slurm.conf and gres.conf files? Kind regards, Joel
Created attachment 46022 [details] Current slurm.conf
Created attachment 46023 [details] Current topology.conf
Created attachment 46024 [details] Current gres.conf for GPU nodes.
Yup. I've added them. -Paul Edmon- On 8/12/26 2:01 AM, bugs@schedmd.com wrote: > Joel Criado <mailto:jcriadoledes@nvidia.com> changed ticket 25687 > <https://support.schedmd.com/show_bug.cgi?id=25687> > What Removed Added > Assignee support@schedmd.com jcriadoledes@nvidia.com > > *Comment # 1 <https://support.schedmd.com/show_bug.cgi?id=25687#c1> on > ticket 25687 <https://support.schedmd.com/show_bug.cgi?id=25687> from > Joel Criado <mailto:jcriadoledes@nvidia.com> * > Hi, > > Can you share your slurm.conf and gres.conf files? > > Kind regards, > Joel > ------------------------------------------------------------------------ > You are receiving this mail because: > > * You reported the ticket. >
Hi, The error is triggered from the function mentioned there: gres_select_filter_select_and_set. There, we try to assign specific GPUs from the selected nodes to the job. The error is emitted when we can't get enough GPUs to fulfill the GRES per job requested. When that happens, the job remains pending and will be tested at the next scheduling cycle. It is not a fatal error for the job, and can end up running later on. From the info on your scontrol show job, it looks like the job had many restarts (11) before, so it is a perfectly runnable job. The error is less harmful than it seems, as you can see the job gets scheduled eventually. I would say that as long as the job gets scheduled in a reasonable time you can safely ignore it. If you see a job pending for a long time and repeated instances of the error we can try to dig further into it. Also, you can change your job request to user GRES per node instead GRES per job. That would mean using --gpus-per-node instead of --gpus. That would trigger a different code path, avoiding the error at all. Kind regards, Joel
Thanks for the explanation. Given this could this be demoted to a warning or given a gate so that it only shows up with you have certain flags on? It doesn't seem to be a critical error at least. -Paul Edmon- On 8/17/2026 10:04 AM, bugs@schedmd.com wrote: > > *Comment # 6 <https://support.schedmd.com/show_bug.cgi?id=25687#c6> on > ticket 25687 <https://support.schedmd.com/show_bug.cgi?id=25687> from > Joel Criado <mailto:jcriadoledes@nvidia.com> * > Hi, > > The error is triggered from the function mentioned there: > gres_select_filter_select_and_set. There, we try to assign specific GPUs from > the selected nodes to the job. The error is emitted when we can't get enough > GPUs to fulfill the GRES per job requested. When that happens, the job remains > pending and will be tested at the next scheduling cycle. It is not a fatal > error for the job, and can end up running later on. > > From the info on your scontrol show job, it looks like the job had many > restarts (11) before, so it is a perfectly runnable job. The error is less > harmful than it seems, as you can see the job gets scheduled eventually. I > would say that as long as the job gets scheduled in a reasonable time you can > safely ignore it. If you see a job pending for a long time and repeated > instances of the error we can try to dig further into it. > > Also, you can change your job request to user GRES per node instead GRES per > job. That would mean using --gpus-per-node instead of --gpus. That would > trigger a different code path, avoiding the error at all. > > Kind regards, > Joel > ------------------------------------------------------------------------ > You are receiving this mail because: > > * You reported the ticket. >
Hi, I will discuss the potential change with some colleagues and let you know what we decide about that. Kind regards, Joel
Thanks! -Paul Edmon- On 8/17/2026 11:02 AM, bugs@schedmd.com wrote: > > *Comment # 8 <https://support.schedmd.com/show_bug.cgi?id=25687#c8> on > ticket 25687 <https://support.schedmd.com/show_bug.cgi?id=25687> from > Joel Criado <mailto:jcriadoledes@nvidia.com> * > Hi, > > I will discuss the potential change with some colleagues and let you know what > we decide about that. > > Kind regards, > Joel > ------------------------------------------------------------------------ > You are receiving this mail because: > > * You reported the ticket. >
