Ticket 17033

We're looking at integrating NVIDIA's DCGM GPU reporting into our Slurm setup per the instructions in this blog post (https://developer.nvidia.com/blog/job-statistics-nvidia-data-center-gpu-manager-slurm/) which boils down to a prolog and epilog script.

Prolog:


# DCGM job statistics
group=$(sudo -u $SLURM_JOB_USER dcgmi group -c allgpus --default)
if [ $? -eq 0 ]; then
  groupid=$(echo $group | awk '{print $10}')
  sudo -u $SLURM_JOB_USER dcgmi stats -g $groupid -e
  sudo -u $SLURM_JOB_USER dcgmi stats -g $groupid -s $SLURM_JOBID
fi

Epilog:

# DCGM job statistics
OUTPUTDIR=$(scontrol show job $SLURM_JOBID | grep WorkDir | cut -d = -f 2)
sudo -u $SLURM_JOB_USER dcgmi stats -x $SLURM_JOBID
sudo -u $SLURM_JOB_USER dcgmi stats -v -j $SLURM_JOBID | \
sudo -u $SLURM_JOB_USER tee $OUTPUTDIR/dcgm-gpu-stats-$HOSTNAME-$SLURM_JOBID.out

We implemented this using the PrologSlurmctld and EpilogSlurmctld parameters in slurm.conf, however the Epilog script seems to run into some issues with the Slurm user using sudo. When logging errors from the commands in the epilog, it looks like the SlurmUser is having issues using sudo to run commands as the job user:

We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

    #1) Respect the privacy of others.
    #2) Think before you type.
    #3) With great power comes great responsibility.

sudo: no tty present and no askpass program specified

Interestingly this doesn't seem to happen with the sudo commands in the prolog. Is this happening because the job has already ended by the time the epilog script runs, and the SlurmUser is thus unable to run commands as the job user anymore?

I would be incredibly grateful for any assistance on getting this working!