SchedMD - Slurm Support – Full Text Ticket Listing
- Home
- | New
- | Browse
- | Search
- | [?]
- | Reports
- | Help
- | New Account
- | Log In
- | Forgot Password
| Summary: | Segfault on job start due to NULL job->job_resrcs | ||
|---|---|---|---|
| Product: | Slurm | Reporter: | John Morrissey <jwm> |
| Component: | slurmctld | Assignee: | Moe Jette <jette> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | 3 - Medium Impact | ||
| Priority: | --- | CC: | dauble |
| Version: | 2.6.x | ||
| Hardware: | Linux | ||
| OS: | Linux | ||
| Site: | Harvard University | Slinky Site: | --- |
| Alineos Sites: | --- | Atos/Eviden Sites: | --- |
| Confidential Site: | --- | Coreweave sites: | --- |
| Cray Sites: | --- | DS9 clusters: | --- |
| Google sites: | --- | HPCnow Sites: | --- |
| HPE Sites: | --- | IBM Sites: | --- |
| NOAA SIte: | --- | NoveTech Sites: | --- |
| Nvidia HWinf-CS Sites: | --- | OCF Sites: | --- |
| Recursion Pharma Sites: | --- | SFW Sites: | --- |
| SNIC sites: | --- | Tzag Elita Sites: | --- |
| Linux Distro: | --- | Machine Name: | |
| CLE Version: | Version Fixed: | ||
| Target Release: | --- | DevPrio: | --- |
| Emory-Cloud Sites: | --- | ||
| Attachments: |
slurmctld logs since process start
likely fix for the problem |
||
|
Description
John Morrissey
2013-10-19 04:03:06 MDT
Created attachment 457 [details]
slurmctld logs since process start
I am not certain exactly what the problem is, but suspect it to be some race condition related to job preemption (from your log): Oct 18 21:52:32 holy-slurm01 slurmctld[39785]: _slurm_rpc_submit_batch_job JobId=2336227 usec=2121609 Oct 18 22:38:22 holy-slurm01 slurmctld[39785]: sched: Allocate JobId=2336227 NodeList=holy2b05108 #CPUs=2 Oct 18 22:38:32 holy-slurm01 slurmctld[39785]: error: find_preemptable_jobs: job 2336227 not pending Oct 18 22:38:32 holy-slurm01 slurmctld[39785]: error: find_preemptable_jobs: job 2336227 not pending Oct 18 22:38:33 holy-slurm01 slurmctld[39785]: error: find_preemptable_jobs: job 2336227 not pending Oct 18 22:38:33 holy-slurm01 slurmctld[39785]: error: find_preemptable_jobs: job 2336227 not pending I will keep looking and update you if I have more information to report. The other thing that I want to raise about your configuration is that you definitely do not want "default_queue_depth=5000". That defines how far down the queue that the scheduler should test every time that a job is submitted or ends (if you did not have defer) or whenever the scheduler runs on other events. There is no relenquishing of locks for other events. The default value is 100 and setting the value to 50000 will result in really poor responsiveness. Let the backfill scheduler do the heavy lifting and it will release locks every couple of seconds for other things to happen. I will add information to the documentation on this matter. Updated documentation:
SchedulerParameters
The interpretation of this parameter varies by SchedulerType.
Multiple options may be comma separated.
default_queue_depth=#
The default number of jobs to attempt scheduling (i.e.
the queue depth) when a running job completes or other
routine actions occur. The full queue will be tested on
a less frequent basis. The default value is 100. In the
case of large clusters (more than 1000 nodes), configur‐
ing a relatively small value may be desirable. Specify‐
ing a large value (say 1000 or higher) can be expected to
result in poor system responsiveness since this schedul‐
ing logic will not release locks for other events to
occur. It would be better to let the backfill scheduler
process a larger number of jobs (see max_job_bf, bf_con‐
tinue and other options here for more information).
defer Setting this option will avoid attempting to schedule
each job individually at job submit time, but defer it
until a later time when scheduling multiple jobs simulta‐
neously may be possible. This option may improve system
responsiveness when large numbers of jobs (many hundreds)
are submitted at the same time, but it will delay the
initiation time of individual jobs. Also see
default_queue_depth above.
bf_continue
The backfill scheduler periodically releases locks in
order to permit other operations to proceed rather than
blocking all activity for what could be an extended
period of time. Setting this option will cause the back‐
fill scheduler to continue processing pending jobs from
its original job list after releasing locks even if job
or node state changes. This can result in lower priority
jobs from being backfill scheduled instead of newly
arrived higher priority jobs, but will permit more queued
jobs to be considered for backfill scheduling.
bf_interval=#
The number of seconds between iterations. Higher values
result in less overhead and better responsiveness. The
default value is 30 seconds. This option applies only to
SchedulerType=sched/backfill.
bf_max_job_part=#
The maximum number of jobs per partition to attempt back‐
fill scheduling for, not counting jobs which cannot be
started due to an association resource limit. This can be
especially helpful for systems with large numbers of par‐
titions and jobs. The default value is 0, which means no
limit. This option applies only to Scheduler‐
Type=sched/backfill.
bf_max_job_user=#
The maximum number of jobs per user to attempt backfill
scheduling for, not counting jobs which cannot be started
due to an association resource limit. One can set this
limit to prevent users from flooding the backfill queue
with jobs that cannot start and that prevent jobs from
other users to start. This is similar to the MAXIJOB
limit in Maui. The default value is 0, which means no
limit. This option applies only to Scheduler‐
Type=sched/backfill.
bf_resolution=#
The number of seconds in the resolution of data main‐
tained about when jobs begin and end. Higher values
result in less overhead and better responsiveness. The
default value is 60 seconds. This option applies only to
SchedulerType=sched/backfill.
bf_window=#
The number of minutes into the future to look when con‐
sidering jobs to schedule. Higher values result in more
overhead and less responsiveness. The default value is
1440 minutes (one day). A value at least as long as the
highest allowed time limit is generally advisable to pre‐
vent job starvation. In order limit the amount of data
managed by the backfill scheduler, if the value of
bf_window is increased, then it is generally advisable to
also increase bf_resolution. if This option applies only
to SchedulerType=sched/backfill.
max_job_bf=#
The maximum number of jobs to attempt backfill scheduling
for (i.e. the queue depth). Higher values result in more
overhead and less responsiveness. Until an attempt is
made to backfill schedule a job, its expected initiation
time value will not be set. The default value is 50.
This option applies only to SchedulerType=sched/backfill.
max_depend_depth=#
Maximum number of
max_switch_wait=#
Maximum number of seconds that a job can delay execution
waiting for the specified desired switch count. The
default value is 300 seconds.
jobs to test for a circular job depen‐
dency. Stop testing after this number of job dependencies
have been tested. The default value is 10 jobs.
max_switch_wait=#
Maximum number of seconds that a job can delay execution
waiting for the specified desired switch count. The
default value is 300 seconds.
Created attachment 458 [details] likely fix for the problem This should fix the problem. Also see commit here: https://github.com/SchedMD/slurm/commit/ea1b316c60ad2863bdc39bb2610229024bce17fa If the backfill scheduler relinquishes locks and the normal job scheduler starts a job that the backfill scheduler was actively working, the backfill scheduler will try to re-schedule that same job, possibly resulting in an invalid memory reference or other badness. Sorry for all the problems here. The bf_continue option is newly added and there are a lot of things that can hapen with the scheduler relinquishes locks. Besides this patch, I recommend changing default_queue_depth to something much smaller, like 100, otherwise performance will be bed max_job_bf to your queue depth, say 50000 bf_max_job_part, I'm not sure what would make sense for you, perhaps 100 bf_resolution=300 to reduce backfill scheduler overhead bf_continue should be safe now Perhaps you could send me some of your slurmdctld log after patching and changes to configuration recommended above. I'm going to close this based upon the patch. Please re-open if necessary. We should tag version 2.6.4 within a couple of weeks and that contains a number of bug fixes specifically for your environment. |