With the following version of slurm, when I use srun for an interactive job, the job is terminated after 62 seconds. I have noticed that no matter what am I doing (leaving the terminal as idle or working with interactive commands), the termination happens. Please see the following output [mahmood@rocks7 ~]$ sinfo -O version VERSION 17.11 [mahmood@rocks7 ~]$ srun -vvvvv -A y8 -p RUBY --ntasks=2 --mem=8GB --pty bash srun: defined options for program `srun' srun: --------------- --------------------- srun: user : `mahmood' srun: uid : 1000 srun: gid : 1000 srun: cwd : /home/mahmood srun: ntasks : 2 (set) srun: nodes : 1 (default) srun: jobid : 4294967294 (default) srun: partition : RUBY srun: profile : `NotSet' srun: job name : `bash' srun: reservation : `(null)' srun: burst_buffer : `(null)' srun: wckey : `(null)' srun: cpu_freq_min : 4294967294 srun: cpu_freq_max : 4294967294 srun: cpu_freq_gov : 4294967294 srun: switches : -1 srun: wait-for-switches : -1 srun: distribution : unknown srun: cpu-bind : default (0) srun: mem-bind : default (0) srun: verbose : 5 srun: slurmd_debug : 0 srun: immediate : false srun: label output : false srun: unbuffered IO : true srun: overcommit : false srun: threads : 60 srun: checkpoint_dir : /var/spool/slurm.checkpoint srun: wait : 60 srun: nice : -2 srun: account : y8 srun: comment : (null) srun: dependency : (null) srun: exclusive : false srun: bcast : false srun: qos : (null) srun: constraints : mem-per-node=8192M srun: geometry : (null) srun: reboot : yes srun: rotate : no srun: preserve_env : false srun: network : (null) srun: propagate : NONE srun: prolog : (null) srun: epilog : (null) srun: mail_type : NONE srun: mail_user : (null) srun: task_prolog : (null) srun: task_epilog : (null) srun: multi_prog : no srun: sockets-per-node : -2 srun: cores-per-socket : -2 srun: threads-per-core : -2 srun: ntasks-per-node : -2 srun: ntasks-per-socket : -2 srun: ntasks-per-core : -2 srun: plane_size : 4294967294 srun: core-spec : NA srun: power : srun: remote command : `bash' srun: debug: propagating RLIMIT_CPU=18446744073709551615 srun: debug: propagating RLIMIT_FSIZE=18446744073709551615 srun: debug: propagating RLIMIT_DATA=18446744073709551615 srun: debug: propagating RLIMIT_STACK=8388608 srun: debug: propagating RLIMIT_CORE=0 srun: debug: propagating RLIMIT_RSS=18446744073709551615 srun: debug: propagating RLIMIT_NPROC=4096 srun: debug: propagating RLIMIT_NOFILE=1024 srun: debug: propagating RLIMIT_MEMLOCK=18446744073709551615 srun: debug: propagating RLIMIT_AS=18446744073709551615 srun: debug: propagating SLURM_PRIO_PROCESS=0 srun: debug: propagating UMASK=0002 srun: debug2: srun PMI messages to port=42250 srun: debug: Entering slurm_allocation_msg_thr_create() srun: debug: port from net_stream_listen is 39781 srun: debug: Entering _msg_thr_internal srun: debug4: eio: handling events for 1 objects srun: debug3: Called eio_message_socket_readable 0 4 srun: debug3: Trying to load plugin /usr/lib64/slurm/auth_munge.so srun: debug: Munge authentication plugin loaded srun: debug3: Success. srun: jobid 304: nodes(1):`compute-0-6', cpu counts: 2(x1) srun: debug2: creating job with 2 tasks srun: debug: requesting job 304, user 1000, nodes 1 including ((null)) srun: debug: cpus 2, tasks 2, name bash, relative 65534 srun: debug2: winsize 39:120 srun: debug2: initialized job control port 34844 srun: debug2: spank: x11.so: local_user_init = 0 srun: debug: Entering slurm_step_launch srun: debug: mpi type = (null) srun: debug: Using mpi/none srun: debug: Entering _msg_thr_create() srun: debug4: eio: handling events for 2 objects srun: debug3: Called eio_message_socket_readable 0 13 srun: debug3: Called eio_message_socket_readable 0 7 srun: debug: initialized stdio listening socket, port 39442 srun: debug: Started IO server thread (140689535534848) srun: debug: Entering _launch_tasks srun: debug3: IO thread pid = 4556 srun: debug4: eio: handling events for 4 objects srun: debug2: Called _file_readable srun: debug3: false, all ioservers not yet initialized srun: debug2: Called _file_writable srun: debug3: false srun: debug3: eof is false srun: debug2: Called _file_writable srun: debug3: false srun: debug3: eof is false srun: debug3: Called _listening_socket_readable srun: launching 304.0 on host compute-0-6, 2 tasks: [0-1] srun: debug3: uid:1000 gid:1000 cwd:/home/mahmood 0 srun: debug3: Trying to load plugin /usr/lib64/slurm/route_default.so srun: route default plugin loaded srun: debug3: Success. srun: debug2: Tree head got back 0 looking for 1 srun: debug3: Tree sending to compute-0-6 srun: debug4: orig_timeout was 20000 we have 0 steps and a timeout of 20000 srun: debug2: Tree head got back 1 srun: debug: launch returned msg_rc=0 err=0 type=8001 srun: debug2: waiting for SIGWINCH srun: debug3: Called _listening_socket_read srun: debug2: Activity on IO listening socket 16 srun: debug3: Accepted IO connection: ip=10.1.1.248 sd=17 srun: debug2: Entering io_init_msg_read_from_fd srun: debug3: read 26 bytes srun: debug2: Leaving io_init_msg_read_from_fd srun: debug2: Entering io_init_msg_validate srun: debug3: msg->version = b001 srun: debug3: msg->nodeid = 0 srun: debug2: Leaving io_init_msg_validate srun: debug2: Validated IO connection from 10.1.1.248, node rank 0, sd=17 srun: debug3: msg.stdout_objs = 1 srun: debug3: msg.stderr_objs = 0 srun: debug3: Called eio_msg_socket_accept srun: debug2: eio_message_socket_accept: got message connection from 10.1.1.248:38204 18 srun: debug2: received task launch srun: Node compute-0-6, 2 tasks started srun: debug4: eio: handling events for 5 objects srun: debug2: Called _file_readable srun: debug2: Called _file_writable srun: debug3: false srun: debug3: eof is false srun: debug2: Called _file_writable srun: debug3: false srun: debug3: eof is false srun: debug3: Called _listening_socket_readable srun: debug4: Called _server_writable srun: debug4: false srun: debug4: Called _server_readable srun: debug4: remote_stdout_objs = 1 srun: debug4: remote_stderr_objs = 0 srun: debug3: task_state_update: step=304.0 task_id=0, TS_START_SUCCESS srun: debug3: task_state_update: step=304.0 task_id=1, TS_START_SUCCESS srun: debug4: eio: handling events for 2 objects srun: debug3: Called eio_message_socket_readable 0 13 srun: debug3: Called eio_message_socket_readable 0 7 srun: debug4: Entering _server_read srun: debug3: Entering io_hdr_read_fd srun: debug3: Leaving io_hdr_read_fd srun: debug4: eio: handling events for 5 objects srun: debug2: Called _file_readable srun: debug2: Called _file_writable srun: debug2: Called _file_writable srun: debug3: false srun: debug3: eof is false srun: debug3: Called _listening_socket_readable srun: debug4: Called _server_writable srun: debug4: false srun: debug4: Called _server_readable srun: debug4: remote_stdout_objs = 1 srun: debug4: remote_stderr_objs = 0 srun: debug2: Entering _file_write [mahmood@compute-0-6 ~]$ srun: debug3: wrote 54 bytes srun: debug2: Leaving _file_write srun: debug4: eio: handling events for 5 objects srun: debug2: Called _file_readable srun: debug2: Called _file_writable srun: debug3: false srun: debug3: eof is false srun: debug2: Called _file_writable srun: debug3: false srun: debug3: eof is false srun: debug3: Called _listening_socket_readable srun: debug4: Called _server_writable srun: debug4: false srun: debug4: Called _server_readable srun: debug4: remote_stdout_objs = 1 srun: debug4: remote_stderr_objs = 0 srun: debug3: Called eio_msg_socket_accept srun: debug2: eio_message_socket_accept: got message connection from 10.1.1.248:38206 18 srun: debug2: received task exit srun: Received task exit notification for 1 task of step 304.0 (status=0x0000). srun: compute-0-6: task 1: Completed srun: debug3: task_state_update: step=304.0 task_id=1, TS_NORMAL_EXIT srun: First task exited. Terminating job in 60s srun: debug: task 1 done srun: debug4: eio: handling events for 2 objects srun: debug3: Called eio_message_socket_readable 0 13 srun: debug3: Called eio_message_socket_readable 0 7 srun: debug3: Called eio_msg_socket_accept srun: debug2: eio_message_socket_accept: got message connection from 10.1.1.1:49284 18 srun: debug3: received ping message srun: debug4: eio: handling events for 1 objects srun: debug3: Called eio_message_socket_readable 0 4 srun: debug3: Called eio_msg_socket_accept srun: debug2: eio_message_socket_accept: got message connection from 10.1.1.1:49288 18 srun: debug3: received ping message srun: debug4: eio: handling events for 1 objects srun: debug3: Called eio_message_socket_readable 0 4 srun: debug3: Called eio_msg_socket_accept srun: debug2: eio_message_socket_accept: got message connection from 10.1.1.1:49294 18 srun: debug3: received ping message srun: debug4: eio: handling events for 1 objects srun: debug3: Called eio_message_socket_readable 0 4 srun: debug3: Called eio_msg_socket_accept srun: debug2: eio_message_socket_accept: got message connection from 10.1.1.1:49296 18 srun: debug3: received ping message srun: debug4: eio: handling events for 1 objects srun: debug3: Called eio_message_socket_readable 0 4 srun: debug3: Called eio_msg_socket_accept srun: debug2: eio_message_socket_accept: got message connection from 10.1.1.1:49300 18 srun: debug3: received ping message srun: debug4: eio: handling events for 1 objects srun: debug3: Called eio_message_socket_readable 0 4 srun: debug3: Called eio_msg_socket_accept srun: debug2: eio_message_socket_accept: got message connection from 10.1.1.1:49306 18 srun: debug3: received ping message srun: debug4: eio: handling events for 1 objects srun: debug3: Called eio_message_socket_readable 0 4 srun: First task exited 60s ago srun: step:304.0 task 0: running srun: step:304.0 task 1: exited srun: Terminating job step 304.0 srun: debug3: Called eio_msg_socket_accept srun: debug2: eio_message_socket_accept: got message connection from 10.1.1.1:57832 20 srun: debug2: received job step complete message srun: Complete job step 304.0 received srun: debug4: eio: handling events for 2 objects srun: debug3: Called eio_message_socket_readable 0 13 srun: debug3: Called eio_message_socket_readable 0 7 srun: Job step aborted: Waiting up to 62 seconds for job step to finish. srun: debug3: Called eio_msg_socket_accept srun: debug2: eio_message_socket_accept: got message connection from 10.1.1.1:57838 18 srun: debug2: received job step complete message srun: Complete job step 304.0 received srun: debug4: eio: handling events for 2 objects srun: debug3: Called eio_message_socket_readable 0 13 srun: debug3: Called eio_message_socket_readable 0 7 srun: debug4: Entering _server_read srun: debug3: Entering io_hdr_read_fd srun: debug3: Leaving io_hdr_read_fd srun: debug3: got eof-stdout msg on _server_read header srun: debug4: eio: handling events for 5 objects srun: debug2: Called _file_readable srun: debug2: Called _file_writable srun: debug3: false srun: debug3: eof is false srun: debug2: Called _file_writable srun: debug3: false srun: debug3: eof is false srun: debug3: Called _listening_socket_readable srun: debug4: Called _server_writable srun: debug4: false, shutdown srun: debug4: Called _server_readable srun: debug3: false, shutdown srun: debug3: Called eio_msg_socket_accept srun: debug2: eio_message_socket_accept: got message connection from 10.1.1.248:38212 17 srun: debug2: received task exit srun: Received task exit notification for 1 task of step 304.0 (status=0x0009). srun: error: compute-0-6: task 0: Killed srun: debug3: task_state_update: step=304.0 task_id=0, TS_ABNORMAL_EXIT srun: debug: task 0 done srun: debug4: eio: handling events for 2 objects srun: debug3: Called eio_message_socket_readable 0 13 srun: debug3: Called eio_message_socket_readable 0 7 srun: debug4: eio: handling events for 2 objects srun: debug3: Called eio_message_socket_readable 1 13 srun: debug2: false, shutdown srun: debug3: Called eio_message_socket_readable 1 7 srun: debug2: false, shutdown srun: debug4: eio: handling events for 5 objects srun: debug2: Called _file_readable srun: debug3: false, shutdown srun: debug2: Called _file_writable srun: debug3: false srun: debug3: eof is false srun: debug2: Called _file_writable srun: debug3: false srun: debug3: eof is false srun: debug3: Called _listening_socket_readable srun: debug2: false, shutdown srun: debug4: Called _server_writable srun: debug4: false, eof srun: debug4: Called _server_readable srun: debug4: false, eof srun: debug: IO thread exiting srun: debug2: slurm_allocation_msg_thr_destroy: clearing up message thread srun: debug4: eio: handling events for 1 objects srun: debug3: Called eio_message_socket_readable 1 4 srun: debug2: false, shutdown srun: debug: Leaving _msg_thr_internal [mahmood@rocks7 ~]$
Mahmood, Could you please let me know which site you work for? Our system couldn't match your gmail address with a supported site. Once we have the information about which site you work for then we can determine how to route this support request. Jacob