SchedMD - Slurm Support – Full Text Ticket Listing
- Home
- | New
- | Browse
- | Search
- | [?]
- | Reports
- | Help
- | New Account
- | Log In
- | Forgot Password
| Summary: | gres_select_filter_select_and_set error | ||
|---|---|---|---|
| Product: | Slurm | Reporter: | Paul Edmon <pedmon> |
| Component: | slurmctld | Assignee: | Joel Criado <jcriadoledes> |
| Status: | OPEN --- | QA Contact: | |
| Severity: | 4 - Minor Issue | ||
| Priority: | --- | ||
| Version: | 26.05.2 | ||
| Hardware: | Linux | ||
| OS: | Linux | ||
| Site: | Harvard University | Slinky Site: | --- |
| Alineos Sites: | --- | Atos/Eviden Sites: | --- |
| Confidential Site: | --- | Coreweave sites: | --- |
| Cray Sites: | --- | DS9 clusters: | --- |
| Google sites: | --- | HPCnow Sites: | --- |
| HPE Sites: | --- | IBM Sites: | --- |
| NOAA SIte: | --- | NoveTech Sites: | --- |
| Nvidia HWinf-CS Sites: | --- | OCF Sites: | --- |
| Recursion Pharma Sites: | --- | SFW Sites: | --- |
| SNIC sites: | --- | Tzag Elita Sites: | --- |
| Linux Distro: | --- | Machine Name: | |
| CLE Version: | Version Fixed: | ||
| Target Release: | --- | DevPrio: | --- |
| Emory-Cloud Sites: | --- | ||
| Attachments: |
Current slurm.conf
Current topology.conf Current gres.conf for GPU nodes. |
||
|
Description
Paul Edmon
2026-08-11 12:30:53 MDT
Hi, Can you share your slurm.conf and gres.conf files? Kind regards, Joel Created attachment 46022 [details]
Current slurm.conf
Created attachment 46023 [details]
Current topology.conf
Created attachment 46024 [details]
Current gres.conf for GPU nodes.
Yup. I've added them. -Paul Edmon- On 8/12/26 2:01 AM, bugs@schedmd.com wrote: > Joel Criado <mailto:jcriadoledes@nvidia.com> changed ticket 25687 > <https://support.schedmd.com/show_bug.cgi?id=25687> > What Removed Added > Assignee support@schedmd.com jcriadoledes@nvidia.com > > *Comment # 1 <https://support.schedmd.com/show_bug.cgi?id=25687#c1> on > ticket 25687 <https://support.schedmd.com/show_bug.cgi?id=25687> from > Joel Criado <mailto:jcriadoledes@nvidia.com> * > Hi, > > Can you share your slurm.conf and gres.conf files? > > Kind regards, > Joel > ------------------------------------------------------------------------ > You are receiving this mail because: > > * You reported the ticket. > Hi, The error is triggered from the function mentioned there: gres_select_filter_select_and_set. There, we try to assign specific GPUs from the selected nodes to the job. The error is emitted when we can't get enough GPUs to fulfill the GRES per job requested. When that happens, the job remains pending and will be tested at the next scheduling cycle. It is not a fatal error for the job, and can end up running later on. From the info on your scontrol show job, it looks like the job had many restarts (11) before, so it is a perfectly runnable job. The error is less harmful than it seems, as you can see the job gets scheduled eventually. I would say that as long as the job gets scheduled in a reasonable time you can safely ignore it. If you see a job pending for a long time and repeated instances of the error we can try to dig further into it. Also, you can change your job request to user GRES per node instead GRES per job. That would mean using --gpus-per-node instead of --gpus. That would trigger a different code path, avoiding the error at all. Kind regards, Joel Thanks for the explanation. Given this could this be demoted to a warning or given a gate so that it only shows up with you have certain flags on? It doesn't seem to be a critical error at least. -Paul Edmon- On 8/17/2026 10:04 AM, bugs@schedmd.com wrote: > > *Comment # 6 <https://support.schedmd.com/show_bug.cgi?id=25687#c6> on > ticket 25687 <https://support.schedmd.com/show_bug.cgi?id=25687> from > Joel Criado <mailto:jcriadoledes@nvidia.com> * > Hi, > > The error is triggered from the function mentioned there: > gres_select_filter_select_and_set. There, we try to assign specific GPUs from > the selected nodes to the job. The error is emitted when we can't get enough > GPUs to fulfill the GRES per job requested. When that happens, the job remains > pending and will be tested at the next scheduling cycle. It is not a fatal > error for the job, and can end up running later on. > > From the info on your scontrol show job, it looks like the job had many > restarts (11) before, so it is a perfectly runnable job. The error is less > harmful than it seems, as you can see the job gets scheduled eventually. I > would say that as long as the job gets scheduled in a reasonable time you can > safely ignore it. If you see a job pending for a long time and repeated > instances of the error we can try to dig further into it. > > Also, you can change your job request to user GRES per node instead GRES per > job. That would mean using --gpus-per-node instead of --gpus. That would > trigger a different code path, avoiding the error at all. > > Kind regards, > Joel > ------------------------------------------------------------------------ > You are receiving this mail because: > > * You reported the ticket. > Hi, I will discuss the potential change with some colleagues and let you know what we decide about that. Kind regards, Joel Thanks! -Paul Edmon- On 8/17/2026 11:02 AM, bugs@schedmd.com wrote: > > *Comment # 8 <https://support.schedmd.com/show_bug.cgi?id=25687#c8> on > ticket 25687 <https://support.schedmd.com/show_bug.cgi?id=25687> from > Joel Criado <mailto:jcriadoledes@nvidia.com> * > Hi, > > I will discuss the potential change with some colleagues and let you know what > we decide about that. > > Kind regards, > Joel > ------------------------------------------------------------------------ > You are receiving this mail because: > > * You reported the ticket. > |