Ticket 14288

Summary: redirection stdout and python time.sleep
Product: Slurm Reporter: Soufiane Belharbi <soufiane.belharbi>
Component: OtherAssignee: Jacob Jenson <jacob>
Status: RESOLVED INVALID QA Contact:
Severity: 6 - No support contract    
Priority: ---    
Version: 21.08.8   
Hardware: Linux   
OS: Linux   
Site: -Other- Slinky Site: ---
Alineos Sites: --- Atos/Eviden Sites: ---
Confidential Site: --- Coreweave sites: ---
Cray Sites: --- DS9 clusters: ---
Google sites: --- HPCnow Sites: ---
HPE Sites: --- IBM Sites: ---
NOAA SIte: --- NoveTech Sites: ---
Nvidia HWinf-CS Sites: --- OCF Sites: ---
Recursion Pharma Sites: --- SFW Sites: ---
SNIC sites: --- Tzag Elita Sites: ---
Linux Distro: --- Machine Name:
CLE Version: Version Fixed:
Target Release: --- DevPrio: ---
Emory-Cloud Sites: ---

Description Soufiane Belharbi 2022-06-09 08:21:20 MDT 
hi,

when submitting this job to slurm, the python code does not write in stdout redirected file as expected.

slurm file:

#!/bin/bash     
#SBATCH --output=ABSOLUTE_PATH_%J.o
#SBATCH --error=ABSOLUTE_PATH_%J.e
#SBATCH --tasks-per-node=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=12M 
#SBATCH --time=00:03:00

python test.py


python file test.py content:

import time

if __name__ == "__main__":
    print('looping!')
    i = 0
    while True:
        print(f'looping {i}')
        i += 1
        time.sleep(1)

- results:
* the redirected file contains everything that comes from bash.
* all prints from python file fails to be logged into the redirected file.

after several trials, it turns out the reason for python to fail to log in the expected redirected file is the line "time.sleep(x)".
when that line is removed, slurm successfully writes what python is printing.

this is not an expected behavior.
any idea why this happens?

thanks
Comment 1 Soufiane Belharbi 2022-06-09 08:48:09 MDT 
p.s.
in addition, when changing python code to the following, the issue happens again. python does not write anything to the redirected file. somehow, the loop without any print in it does not trigger flushing the text to the redirected file.

also, slurm does not flush text even after the job is done.

content of test.py:

import time


if __name__ == "__main__":
    print('looping!')
    i = 0
    while True:
        i = 1


thanks