Ticket 2661

Summary: Missing output file
Product: Slurm Reporter: Davide Vanzo <davide.vanzo>
Component: SchedulingAssignee: Alejandro Sanchez <asanchezg>
Status: RESOLVED INFOGIVEN QA Contact:
Severity: 4 - Minor Issue    
Priority: --- CC: asanchezg
Version: 15.08.8   
Hardware: Linux   
OS: Linux   
Site: Vanderbilt Slinky Site: ---
Alineos Sites: --- Atos/Eviden Sites: ---
Confidential Site: --- Coreweave sites: ---
Cray Sites: --- DS9 clusters: ---
Google sites: --- HPCnow Sites: ---
HPE Sites: --- IBM Sites: ---
NOAA SIte: --- NoveTech Sites: ---
Nvidia HWinf-CS Sites: --- OCF Sites: ---
Recursion Pharma Sites: --- SFW Sites: ---
SNIC sites: --- Tzag Elita Sites: ---
Linux Distro: --- Machine Name:
CLE Version: Version Fixed:
Target Release: --- DevPrio: ---
Emory-Cloud Sites: ---

Description Davide Vanzo 2016-04-25 04:59:37 MDT 
We some users complaining that the output files (--output and --error option) of failed jobs within a job array are missing. This makes the identification of the cause of the failure impossible since the only thing known is the non-zero exit code. Any idea?

Davide
Comment 1 Alejandro Sanchez 2016-04-26 02:44:43 MDT 
Hello Davide,

I don't see this problem. Tried to reproduce with both 15.08.8 and 15.08.10 and in either case error and output files are generated:

alex@pc:~/t$ cat script.batch 
#!/bin/bash
#SBATCH -o /home/alex/t/%j.out
#SBATCH -e /home/alex/t/%j.err
srun nonsense
alex@pc:~/t$ sbatch --array=0-1 script.batch 
Submitted batch job 20013
alex@pc:~/t$ ls -ltr 20013.* 20014.*
-rw-rw-r-- 1 alex alex   0 abr 26 17:39 20014.out
-rw-rw-r-- 1 alex alex   0 abr 26 17:39 20013.out
-rw-rw-r-- 1 alex alex 122 abr 26 17:39 20014.err
-rw-rw-r-- 1 alex alex 122 abr 26 17:39 20013.err
alex@pc:~/t$ pwd
/home/alex/t
alex@pc:~/t$ cat 20014.err
srun: error: compute1: task 0: Exited with exit code 2
slurmstepd-compute1: execve(): nonsense: No such file or directory
alex@pc:~/t$ cat 20013.err
srun: error: compute2: task 0: Exited with exit code 2
slurmstepd-compute2: execve(): nonsense: No such file or directory
alex@pc:~/t$

Could you please ensure whether:

1. Users complaining about missing --output/--error files are not doing something odd with -e/-o options?
2. Isn't there any problem with the nodes and specially with the filesystem where files are supposed to be created?

Thanks.
Comment 2 Alejandro Sanchez 2016-05-09 01:23:11 MDT 
Hi Davide. Any updates on this? Thanks.
Comment 3 Davide Vanzo 2016-05-09 01:25:32 MDT 
Alejandro,
unfortunately the feedback from the user was insufficient so I needed his job data to try to replicate it. I just received everything and I'll keep you posted on this.
Thanks for your help and patience.

Davide
Comment 4 Alejandro Sanchez 2016-06-13 23:26:49 MDT 
Hello Davide. Any progress with the user data? Thanks.
Comment 5 Davide Vanzo 2016-06-14 00:40:25 MDT 
Hi Alejandro,
I was not able to replicate the issue the user had, hence the problem seemed to be the user itself. Sorry for the long wait.
Feel free to close the ticket.

Davide
Comment 6 Alejandro Sanchez 2016-06-14 00:49:09 MDT 
No problem. Closing resolved/infogiven. Please reopen if you encounter any more issues. Thanks.