Ticket 5182 - srun command is automatically terminated after 62 seconds
Summary: srun command is automatically terminated after 62 seconds
Status: RESOLVED INVALID
Alias: None
Product: Slurm
Classification: Unclassified
Component: Scheduling (show other tickets)
Version: 17.11.0
Hardware: Linux Linux
: 6 - No support contract
Assignee: Jacob Jenson
QA Contact:
URL:
Depends on:
Blocks:
 
Reported: 2018-05-17 05:18 MDT by Mahmood Naderan
Modified: 2018-05-17 11:24 MDT (History)
0 users

See Also:
Site: -Other-
Slinky Site: ---
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
Google sites: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
NoveTech Sites: ---
Nvidia HWinf-CS Sites: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Tzag Elita Sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed:
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---


Attachments

Note You need to log in before you can comment on or make changes to this ticket.
Description Mahmood Naderan 2018-05-17 05:18:24 MDT
With the following version of slurm, when I use srun for an interactive job, the job is terminated after 62 seconds. I have noticed that no matter what am I doing (leaving the terminal as idle or working with interactive commands), the termination happens.

Please see the following output

[mahmood@rocks7 ~]$ sinfo -O version
VERSION
17.11
[mahmood@rocks7 ~]$ srun -vvvvv -A y8 -p RUBY --ntasks=2 --mem=8GB --pty bash
srun: defined options for program `srun'
srun: --------------- ---------------------
srun: user           : `mahmood'
srun: uid            : 1000
srun: gid            : 1000
srun: cwd            : /home/mahmood
srun: ntasks         : 2 (set)
srun: nodes          : 1 (default)
srun: jobid          : 4294967294 (default)
srun: partition      : RUBY
srun: profile        : `NotSet'
srun: job name       : `bash'
srun: reservation    : `(null)'
srun: burst_buffer   : `(null)'
srun: wckey          : `(null)'
srun: cpu_freq_min   : 4294967294
srun: cpu_freq_max   : 4294967294
srun: cpu_freq_gov   : 4294967294
srun: switches       : -1
srun: wait-for-switches : -1
srun: distribution   : unknown
srun: cpu-bind       : default (0)
srun: mem-bind       : default (0)
srun: verbose        : 5
srun: slurmd_debug   : 0
srun: immediate      : false
srun: label output   : false
srun: unbuffered IO  : true
srun: overcommit     : false
srun: threads        : 60
srun: checkpoint_dir : /var/spool/slurm.checkpoint
srun: wait           : 60
srun: nice           : -2
srun: account        : y8
srun: comment        : (null)
srun: dependency     : (null)
srun: exclusive      : false
srun: bcast          : false
srun: qos            : (null)
srun: constraints    : mem-per-node=8192M
srun: geometry       : (null)
srun: reboot         : yes
srun: rotate         : no
srun: preserve_env   : false
srun: network        : (null)
srun: propagate      : NONE
srun: prolog         : (null)
srun: epilog         : (null)
srun: mail_type      : NONE
srun: mail_user      : (null)
srun: task_prolog    : (null)
srun: task_epilog    : (null)
srun: multi_prog     : no
srun: sockets-per-node  : -2
srun: cores-per-socket  : -2
srun: threads-per-core  : -2
srun: ntasks-per-node   : -2
srun: ntasks-per-socket : -2
srun: ntasks-per-core   : -2
srun: plane_size        : 4294967294
srun: core-spec         : NA
srun: power             :
srun: remote command    : `bash'
srun: debug:  propagating RLIMIT_CPU=18446744073709551615
srun: debug:  propagating RLIMIT_FSIZE=18446744073709551615
srun: debug:  propagating RLIMIT_DATA=18446744073709551615
srun: debug:  propagating RLIMIT_STACK=8388608
srun: debug:  propagating RLIMIT_CORE=0
srun: debug:  propagating RLIMIT_RSS=18446744073709551615
srun: debug:  propagating RLIMIT_NPROC=4096
srun: debug:  propagating RLIMIT_NOFILE=1024
srun: debug:  propagating RLIMIT_MEMLOCK=18446744073709551615
srun: debug:  propagating RLIMIT_AS=18446744073709551615
srun: debug:  propagating SLURM_PRIO_PROCESS=0
srun: debug:  propagating UMASK=0002
srun: debug2: srun PMI messages to port=42250
srun: debug:  Entering slurm_allocation_msg_thr_create()
srun: debug:  port from net_stream_listen is 39781
srun: debug:  Entering _msg_thr_internal
srun: debug4: eio: handling events for 1 objects
srun: debug3: Called eio_message_socket_readable 0 4
srun: debug3: Trying to load plugin /usr/lib64/slurm/auth_munge.so
srun: debug:  Munge authentication plugin loaded
srun: debug3: Success.
srun: jobid 304: nodes(1):`compute-0-6', cpu counts: 2(x1)
srun: debug2: creating job with 2 tasks
srun: debug:  requesting job 304, user 1000, nodes 1 including ((null))
srun: debug:  cpus 2, tasks 2, name bash, relative 65534
srun: debug2: winsize 39:120
srun: debug2: initialized job control port 34844
srun: debug2: spank: x11.so: local_user_init = 0
srun: debug:  Entering slurm_step_launch
srun: debug:  mpi type = (null)
srun: debug:  Using mpi/none
srun: debug:  Entering _msg_thr_create()
srun: debug4: eio: handling events for 2 objects
srun: debug3: Called eio_message_socket_readable 0 13
srun: debug3: Called eio_message_socket_readable 0 7
srun: debug:  initialized stdio listening socket, port 39442
srun: debug:  Started IO server thread (140689535534848)
srun: debug:  Entering _launch_tasks
srun: debug3: IO thread pid = 4556
srun: debug4: eio: handling events for 4 objects
srun: debug2: Called _file_readable
srun: debug3:   false, all ioservers not yet initialized
srun: debug2: Called _file_writable
srun: debug3:   false
srun: debug3:   eof is false
srun: debug2: Called _file_writable
srun: debug3:   false
srun: debug3:   eof is false
srun: debug3: Called _listening_socket_readable
srun: launching 304.0 on host compute-0-6, 2 tasks: [0-1]
srun: debug3: uid:1000 gid:1000 cwd:/home/mahmood 0
srun: debug3: Trying to load plugin /usr/lib64/slurm/route_default.so
srun: route default plugin loaded
srun: debug3: Success.
srun: debug2: Tree head got back 0 looking for 1
srun: debug3: Tree sending to compute-0-6
srun: debug4: orig_timeout was 20000 we have 0 steps and a timeout of 20000
srun: debug2: Tree head got back 1
srun: debug:  launch returned msg_rc=0 err=0 type=8001
srun: debug2: waiting for SIGWINCH
srun: debug3: Called _listening_socket_read
srun: debug2: Activity on IO listening socket 16
srun: debug3: Accepted IO connection: ip=10.1.1.248 sd=17
srun: debug2: Entering io_init_msg_read_from_fd
srun: debug3:   read 26 bytes
srun: debug2: Leaving  io_init_msg_read_from_fd
srun: debug2: Entering io_init_msg_validate
srun: debug3:   msg->version = b001
srun: debug3:   msg->nodeid = 0
srun: debug2: Leaving  io_init_msg_validate
srun: debug2: Validated IO connection from 10.1.1.248, node rank 0, sd=17
srun: debug3: msg.stdout_objs = 1
srun: debug3: msg.stderr_objs = 0
srun: debug3: Called eio_msg_socket_accept
srun: debug2: eio_message_socket_accept: got message connection from 10.1.1.248:38204 18
srun: debug2: received task launch
srun: Node compute-0-6, 2 tasks started
srun: debug4: eio: handling events for 5 objects
srun: debug2: Called _file_readable
srun: debug2: Called _file_writable
srun: debug3:   false
srun: debug3:   eof is false
srun: debug2: Called _file_writable
srun: debug3:   false
srun: debug3:   eof is false
srun: debug3: Called _listening_socket_readable
srun: debug4: Called _server_writable
srun: debug4:   false
srun: debug4: Called _server_readable
srun: debug4: remote_stdout_objs = 1
srun: debug4: remote_stderr_objs = 0
srun: debug3: task_state_update: step=304.0 task_id=0, TS_START_SUCCESS
srun: debug3: task_state_update: step=304.0 task_id=1, TS_START_SUCCESS
srun: debug4: eio: handling events for 2 objects
srun: debug3: Called eio_message_socket_readable 0 13
srun: debug3: Called eio_message_socket_readable 0 7
srun: debug4: Entering _server_read
srun: debug3: Entering io_hdr_read_fd
srun: debug3: Leaving  io_hdr_read_fd
srun: debug4: eio: handling events for 5 objects
srun: debug2: Called _file_readable
srun: debug2: Called _file_writable
srun: debug2: Called _file_writable
srun: debug3:   false
srun: debug3:   eof is false
srun: debug3: Called _listening_socket_readable
srun: debug4: Called _server_writable
srun: debug4:   false
srun: debug4: Called _server_readable
srun: debug4: remote_stdout_objs = 1
srun: debug4: remote_stderr_objs = 0
srun: debug2: Entering _file_write
[mahmood@compute-0-6 ~]$ srun: debug3:   wrote 54 bytes
srun: debug2: Leaving  _file_write
srun: debug4: eio: handling events for 5 objects
srun: debug2: Called _file_readable
srun: debug2: Called _file_writable
srun: debug3:   false
srun: debug3:   eof is false
srun: debug2: Called _file_writable
srun: debug3:   false
srun: debug3:   eof is false
srun: debug3: Called _listening_socket_readable
srun: debug4: Called _server_writable
srun: debug4:   false
srun: debug4: Called _server_readable
srun: debug4: remote_stdout_objs = 1
srun: debug4: remote_stderr_objs = 0
srun: debug3: Called eio_msg_socket_accept
srun: debug2: eio_message_socket_accept: got message connection from 10.1.1.248:38206 18
srun: debug2: received task exit
srun: Received task exit notification for 1 task of step 304.0 (status=0x0000).
srun: compute-0-6: task 1: Completed
srun: debug3: task_state_update: step=304.0 task_id=1, TS_NORMAL_EXIT
srun: First task exited. Terminating job in 60s
srun: debug:  task 1 done
srun: debug4: eio: handling events for 2 objects
srun: debug3: Called eio_message_socket_readable 0 13
srun: debug3: Called eio_message_socket_readable 0 7
srun: debug3: Called eio_msg_socket_accept
srun: debug2: eio_message_socket_accept: got message connection from 10.1.1.1:49284 18
srun: debug3: received ping message
srun: debug4: eio: handling events for 1 objects
srun: debug3: Called eio_message_socket_readable 0 4
srun: debug3: Called eio_msg_socket_accept
srun: debug2: eio_message_socket_accept: got message connection from 10.1.1.1:49288 18
srun: debug3: received ping message
srun: debug4: eio: handling events for 1 objects
srun: debug3: Called eio_message_socket_readable 0 4
srun: debug3: Called eio_msg_socket_accept
srun: debug2: eio_message_socket_accept: got message connection from 10.1.1.1:49294 18
srun: debug3: received ping message
srun: debug4: eio: handling events for 1 objects
srun: debug3: Called eio_message_socket_readable 0 4
srun: debug3: Called eio_msg_socket_accept
srun: debug2: eio_message_socket_accept: got message connection from 10.1.1.1:49296 18
srun: debug3: received ping message
srun: debug4: eio: handling events for 1 objects
srun: debug3: Called eio_message_socket_readable 0 4
srun: debug3: Called eio_msg_socket_accept
srun: debug2: eio_message_socket_accept: got message connection from 10.1.1.1:49300 18
srun: debug3: received ping message
srun: debug4: eio: handling events for 1 objects
srun: debug3: Called eio_message_socket_readable 0 4
srun: debug3: Called eio_msg_socket_accept
srun: debug2: eio_message_socket_accept: got message connection from 10.1.1.1:49306 18
srun: debug3: received ping message
srun: debug4: eio: handling events for 1 objects
srun: debug3: Called eio_message_socket_readable 0 4
srun: First task exited 60s ago
srun: step:304.0 task 0: running
srun: step:304.0 task 1: exited
srun: Terminating job step 304.0
srun: debug3: Called eio_msg_socket_accept
srun: debug2: eio_message_socket_accept: got message connection from 10.1.1.1:57832 20
srun: debug2: received job step complete message
srun: Complete job step 304.0 received
srun: debug4: eio: handling events for 2 objects
srun: debug3: Called eio_message_socket_readable 0 13
srun: debug3: Called eio_message_socket_readable 0 7
srun: Job step aborted: Waiting up to 62 seconds for job step to finish.
srun: debug3: Called eio_msg_socket_accept
srun: debug2: eio_message_socket_accept: got message connection from 10.1.1.1:57838 18
srun: debug2: received job step complete message
srun: Complete job step 304.0 received
srun: debug4: eio: handling events for 2 objects
srun: debug3: Called eio_message_socket_readable 0 13
srun: debug3: Called eio_message_socket_readable 0 7
srun: debug4: Entering _server_read
srun: debug3: Entering io_hdr_read_fd
srun: debug3: Leaving  io_hdr_read_fd
srun: debug3: got eof-stdout msg on _server_read header
srun: debug4: eio: handling events for 5 objects
srun: debug2: Called _file_readable
srun: debug2: Called _file_writable
srun: debug3:   false
srun: debug3:   eof is false
srun: debug2: Called _file_writable
srun: debug3:   false
srun: debug3:   eof is false
srun: debug3: Called _listening_socket_readable
srun: debug4: Called _server_writable
srun: debug4:   false, shutdown
srun: debug4: Called _server_readable
srun: debug3:   false, shutdown
srun: debug3: Called eio_msg_socket_accept
srun: debug2: eio_message_socket_accept: got message connection from 10.1.1.248:38212 17
srun: debug2: received task exit
srun: Received task exit notification for 1 task of step 304.0 (status=0x0009).
srun: error: compute-0-6: task 0: Killed
srun: debug3: task_state_update: step=304.0 task_id=0, TS_ABNORMAL_EXIT
srun: debug:  task 0 done
srun: debug4: eio: handling events for 2 objects
srun: debug3: Called eio_message_socket_readable 0 13
srun: debug3: Called eio_message_socket_readable 0 7
srun: debug4: eio: handling events for 2 objects
srun: debug3: Called eio_message_socket_readable 1 13
srun: debug2:   false, shutdown
srun: debug3: Called eio_message_socket_readable 1 7
srun: debug2:   false, shutdown
srun: debug4: eio: handling events for 5 objects
srun: debug2: Called _file_readable
srun: debug3:   false, shutdown
srun: debug2: Called _file_writable
srun: debug3:   false
srun: debug3:   eof is false
srun: debug2: Called _file_writable
srun: debug3:   false
srun: debug3:   eof is false
srun: debug3: Called _listening_socket_readable
srun: debug2:   false, shutdown
srun: debug4: Called _server_writable
srun: debug4:   false, eof
srun: debug4: Called _server_readable
srun: debug4:   false, eof
srun: debug:  IO thread exiting
srun: debug2: slurm_allocation_msg_thr_destroy: clearing up message thread
srun: debug4: eio: handling events for 1 objects
srun: debug3: Called eio_message_socket_readable 1 4
srun: debug2:   false, shutdown
srun: debug:  Leaving _msg_thr_internal
[mahmood@rocks7 ~]$
Comment 1 Jacob Jenson 2018-05-17 09:22:50 MDT
Mahmood,

Could you please let me know which site you work for? Our system couldn't match your gmail address with a supported site. Once we have the information about which site you work for then we can determine how to route this support request. 

Jacob