Ticket 694

I have found the place in req.c that is killing the job... I think I'll comment it out, rebuild and redeploy.

I think it is ok to comment that code for a now. However what is the problem exactly?

On April 13, 2014 9:12:22 AM PDT, bugs@schedmd.com wrote:
>http://bugs.schedmd.com/show_bug.cgi?id=694
>
>--- Comment #1 from Stuart Midgley <stuartm@dugeo.com> ---
>I have found the place in req.c that is killing the job... I think I'll
>comment
>it out, rebuild and redeploy.
>
>-- 
>You are receiving this mail because:
>You are on the CC list for the bug.
>You are the assignee for the bug.
>You are watching someone on the CC list of the bug.
>You are watching the assignee of the bug.

The issue is that our users don't set their memory requests correctly or accurately.  They set it accurately enough for scheduling purposes but not for job control/monitoring.

I'd like to have the ability to allow the real memory to be over subscribed (like with virtual and time limits etc).

Nearly all jobs on our system use a whole node (--exclusive) so their memory request is sort of irrelavent (except for scheduling).  I'd basically like someway for their jobs to go over their memory request.

A parameter like VSizeFactor (Perhaps RSizeFactor ?)


The other issue is the exit status of the job.  I need to be able to detect all cases where the queue kills a job, catch it in the epilogslurmctld script, mark the job with SpecialExit and reschedule it in held state.

Thanks.

Slurm can easily be configured to completely ignore memory limits. For the SchedulingParameters change "CR_Core_Memory" or "CR_CPU_Memory" to just "CR_Core" or "CR_CPU". That might be the best solution unless or until your users are prepared to set memory limits.

Many sites also configure TaskPlugin=task/cgroup, which creates a container of the exact memory size requested by that job. The job will get malloc failures after reaching that limit. We could still possibly support an "overcommit" type of function that automatically increased each job's specified memory limit by some amount, but that does not exist today.

If we go down the CR_CPOU route, I assume that memory isn't used for scheduling purposes?  

We have a heterogeneous cluster and the memory request is a sort of "minimum" to run the job.  When the program starts up, it looks at the memory available and start an appropriate number of threads.

Extra... this is what I applied

diff --git a/src/common/slurm_jobacct_gather.c b/src/common/slurm_jobacct_gather.c
index 332ab7b80a..bd849d9df9 100644
--- a/src/common/slurm_jobacct_gather.c
+++ b/src/common/slurm_jobacct_gather.c
@@ -558,7 +558,7 @@ extern void jobacct_gather_handle_mem_limit(
                              total_job_mem, jobacct_mem_limit);
                }
        }
-       if (jobacct_job_id && jobacct_mem_limit &&
+/*     if (jobacct_job_id && jobacct_mem_limit &&
            (total_job_mem > jobacct_mem_limit)) {
                if (jobacct_step_id == NO_VAL) {
                        error("Job %u exceeded memory limit "
@@ -587,7 +587,7 @@ extern void jobacct_gather_handle_mem_limit(
                              jobacct_vmem_limit);
                }
                _acct_kill_step();
-       }
+       } */
 }
 
 /********************* jobacctinfo functions ******************************/
diff --git a/src/slurmd/slurmd/req.c b/src/slurmd/slurmd/req.c
index e43599222d..44eb559a34 100644
--- a/src/slurmd/slurmd/req.c
+++ b/src/slurmd/slurmd/req.c
@@ -2136,7 +2136,7 @@ _rpc_ping(slurm_msg_t *msg)
        }
 
        /* Take this opportunity to enforce any job memory limits */
-       _enforce_job_mem_limit();
+       /* _enforce_job_mem_limit(); */
        return rc;
 }
 
@@ -2171,7 +2171,7 @@ _rpc_health_check(slurm_msg_t *msg)
        }
 
        /* Take this opportunity to enforce any job memory limits */
-       _enforce_job_mem_limit();
+       /* _enforce_job_mem_limit(); */
        return rc;
 }




Even if we go with CR_CPU these routines still get executed don't they?  Or is the mem limit set to some massive value somewhere?

As I recall, when memory limits are not enforced the limits are set to zero (i.e. jobacct_mem_limit is zero below):

-       if (jobacct_job_id && jobacct_mem_limit &&

Hi,
   if your users specify --mem-per-cpu than Slurm will enforce it regardless of
CR_CORE or CR_CORE_Memory. We believe for now it is better for you to keep your
code changes that will prevent the termination of jobs.
We plan to develop a new slurm.conf parameter that will not kill jobs if they
use more memory than requested.

David

Thanks.

Any word on the epilogslurmctld issue?

Do you mean exit status after scancel or terminated job?

On April 14, 2014 6:07:09 PM PDT, bugs@schedmd.com wrote:
>http://bugs.schedmd.com/show_bug.cgi?id=694
>
>--- Comment #9 from Stuart Midgley <stuartm@dugeo.com> ---
>Thanks.
>
>Any word on the epilogslurmctld issue?
>
>-- 
>You are receiving this mail because:
>You are on the CC list for the bug.
>You are the assignee for the bug.
>You are watching someone on the CC list of the bug.
>You are watching the assignee of the bug.

Created attachment 760 [details]
attachment-31611-0.html

After a job is terminated by Slurm for some reason (exceeding some limit).  User induced I'm not so worried about.

The sure thing is that exit is not zero. This is something to think about.

On April 14, 2014 6:49:50 PM PDT, bugs@schedmd.com wrote:
>http://bugs.schedmd.com/show_bug.cgi?id=694
>
>--- Comment #12 from Stuart Midgley <stuartm@dugeo.com> ---
>After a job is terminated by Slurm for some reason (exceeding some
>limit). 
>User induced I'm not so worried about.
>
>-- 
>You are receiving this mail because:
>You are on the CC list for the bug.
>You are the assignee for the bug.
>You are watching someone on the CC list of the bug.
>You are watching the assignee of the bug.

Created attachment 761 [details]
attachment-15620-0.html

Hmmm...  unless you do something different with SLURM_JOB_EXIT_CODE2... Upon closer reading of the man page, should I also be looking at the signal sent?  Which I currently ignore.



#!/bin/bash

{

export PATH=/d/sw/slurm/latest/bin:/d/sw/slurm/latest/sbin:$PATH

JOB_EXIT=${SLURM_JOB_EXIT_CODE2%:*}

if (( JOB_EXIT > 0 )); then
    echo "$(date)    "$(for v in ${!SLURM_*}; do echo -n "${v}=${!v} "; done)

    for((i=1; ; i++)); do
        status=0
        scontrol requeuehold state=SpecialExit ${SLURM_JOB_ID} && break
        status=$?
        sleep $(( RANDOM % (30+i) ))
    done

    exit $status
fi

} >> /var/log/slurm/slurmctld_epilog.log 2>&1

This is a little fishy.  I've read the documenation for SLURM_JOB_EXIT_CODE and it idicates it should either be -1 or the PID of the process that finished (according to my reading of man 2 wait)...

now, I've looked in my epilog log and ALL the exit codes are the same (even on different machines)

    SLURM_JOB_EXIT_CODE=25600

Which doesn't seem right.

So, I assume I need to look at the <sig> in SLURM_JOB_EXIT_CODE2?  If so, what values are those?  In the case where Slurm kills jobs for going over memory?

We return what wait() does using the well known macros to get the meaningful bits. I think we should make sure there is no job kill by slurm so all return codes are your business :-)

On April 14, 2014 7:34:31 PM PDT, bugs@schedmd.com wrote:
>http://bugs.schedmd.com/show_bug.cgi?id=694
>
>--- Comment #17 from Stuart Midgley <stuartm@dugeo.com> ---
>So, I assume I need to look at the <sig> in SLURM_JOB_EXIT_CODE2?  If
>so, what
>values are those?  In the case where Slurm kills jobs for going over
>memory?
>
>-- 
>You are receiving this mail because:
>You are on the CC list for the bug.
>You are the assignee for the bug.
>You are watching someone on the CC list of the bug.
>You are watching the assignee of the bug.

Created attachment 763 [details]
attachment-17020-0.html

Fixed in commit 6b5646571cb2. 

David