Ticket 2661 - Missing output file
Summary: Missing output file
Status: RESOLVED INFOGIVEN
Alias: None
Product: Slurm
Classification: Unclassified
Component: Scheduling (show other tickets)
Version: 15.08.8
Hardware: Linux Linux
: 4 - Minor Issue
Assignee: Alejandro Sanchez
QA Contact:
URL:
Depends on:
Blocks:
 
Reported: 2016-04-25 04:59 MDT by Davide Vanzo
Modified: 2021-07-13 03:53 MDT (History)
1 user (show)

See Also:
Site: Vanderbilt
Slinky Site: ---
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
Google sites: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
NoveTech Sites: ---
Nvidia HWinf-CS Sites: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Tzag Elita Sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed:
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---


Attachments

Note You need to log in before you can comment on or make changes to this ticket.
Description Davide Vanzo 2016-04-25 04:59:37 MDT
We some users complaining that the output files (--output and --error option) of failed jobs within a job array are missing. This makes the identification of the cause of the failure impossible since the only thing known is the non-zero exit code. Any idea?

Davide
Comment 1 Alejandro Sanchez 2016-04-26 02:44:43 MDT
Hello Davide,

I don't see this problem. Tried to reproduce with both 15.08.8 and 15.08.10 and in either case error and output files are generated:

alex@pc:~/t$ cat script.batch 
#!/bin/bash
#SBATCH -o /home/alex/t/%j.out
#SBATCH -e /home/alex/t/%j.err
srun nonsense
alex@pc:~/t$ sbatch --array=0-1 script.batch 
Submitted batch job 20013
alex@pc:~/t$ ls -ltr 20013.* 20014.*
-rw-rw-r-- 1 alex alex   0 abr 26 17:39 20014.out
-rw-rw-r-- 1 alex alex   0 abr 26 17:39 20013.out
-rw-rw-r-- 1 alex alex 122 abr 26 17:39 20014.err
-rw-rw-r-- 1 alex alex 122 abr 26 17:39 20013.err
alex@pc:~/t$ pwd
/home/alex/t
alex@pc:~/t$ cat 20014.err
srun: error: compute1: task 0: Exited with exit code 2
slurmstepd-compute1: execve(): nonsense: No such file or directory
alex@pc:~/t$ cat 20013.err
srun: error: compute2: task 0: Exited with exit code 2
slurmstepd-compute2: execve(): nonsense: No such file or directory
alex@pc:~/t$

Could you please ensure whether:

1. Users complaining about missing --output/--error files are not doing something odd with -e/-o options?
2. Isn't there any problem with the nodes and specially with the filesystem where files are supposed to be created?

Thanks.
Comment 2 Alejandro Sanchez 2016-05-09 01:23:11 MDT
Hi Davide. Any updates on this? Thanks.
Comment 3 Davide Vanzo 2016-05-09 01:25:32 MDT
Alejandro,
unfortunately the feedback from the user was insufficient so I needed his job data to try to replicate it. I just received everything and I'll keep you posted on this.
Thanks for your help and patience.

Davide
Comment 4 Alejandro Sanchez 2016-06-13 23:26:49 MDT
Hello Davide. Any progress with the user data? Thanks.
Comment 5 Davide Vanzo 2016-06-14 00:40:25 MDT
Hi Alejandro,
I was not able to replicate the issue the user had, hence the problem seemed to be the user itself. Sorry for the long wait.
Feel free to close the ticket.

Davide
Comment 6 Alejandro Sanchez 2016-06-14 00:49:09 MDT
No problem. Closing resolved/infogiven. Please reopen if you encounter any more issues. Thanks.