Ticket 9514 - scontrol reboot- node stays in drained state after reboot
Summary: scontrol reboot- node stays in drained state after reboot
Status: RESOLVED DUPLICATE of ticket 9513
Alias: None
Product: Slurm
Classification: Unclassified
Component: slurmctld (show other tickets)
Version: 20.02.3
Hardware: Linux Linux
: 4 - Minor Issue
Assignee: Director of Support
QA Contact:
URL:
Depends on:
Blocks:
 
Reported: 2020-08-04 14:32 MDT by lhuang
Modified: 2020-08-04 14:41 MDT (History)
0 users

See Also:
Site: NY Genome
Slinky Site: ---
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
Google sites: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
NoveTech Sites: ---
Nvidia HWinf-CS Sites: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Tzag Elita Sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed:
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this ticket.
Description lhuang 2020-08-04 14:32:35 MDT 
On our dev cluster, we are testing 19.05 upgrade to 20.02.3. After the upgrade we rebooted the nodes but they did not come back from the drained state. I've attached the slurm.conf

scontrol reboot devslurmvm0[1-2]

[2020-08-04T16:25:02.755] Set debug level to 6
[2020-08-04T16:25:14.852] debug2: Processing RPC: REQUEST_REBOOT_NODES from uid=0
[2020-08-04T16:25:14.852] reboot request queued for nodes devslurmvm[01-02]
[2020-08-04T16:25:15.178] debug2: Testing job time limits and checkpoints
[2020-08-04T16:25:15.178] debug:  Queuing reboot request for nodes devslurmvm[01-02]
[2020-08-04T16:25:15.178] debug2: Performing purge of old job records
[2020-08-04T16:25:15.178] SchedulerParameters=default_queue_depth=100,max_rpc_cnt=0,max_sched_time=2,partition_job_depth=0,sched_max_job_start=0,sched_min_interval=2
[2020-08-04T16:25:15.178] debug:  sched: Running job scheduler
[2020-08-04T16:25:15.178] debug2: Spawning RPC agent for msg_type REQUEST_REBOOT_NODES
[2020-08-04T16:25:15.306] debug:  backfill: beginning
[2020-08-04T16:25:15.306] debug:  backfill: no jobs to backfill
[2020-08-04T16:25:16.179] debug:  slurm_send_only_node_msg: poll timed out with 0 outstanding: Resource temporarily unavailable
[2020-08-04T16:25:16.179] agent/is_node_resp: node:devslurmvm02 RPC:REQUEST_REBOOT_NODES : Resource temporarily unavailable
[2020-08-04T16:25:16.179] debug:  slurm_send_only_node_msg: poll timed out with 0 outstanding: Resource temporarily unavailable
[2020-08-04T16:25:16.180] agent/is_node_resp: node:devslurmvm01 RPC:REQUEST_REBOOT_NODES : Resource temporarily unavailable
[2020-08-04T16:25:16.454] debug:  node_not_resp: node devslurmvm01 responded since msg sent
[2020-08-04T16:25:16.454] debug:  node_not_resp: node devslurmvm02 responded since msg sent
[2020-08-04T16:25:45.207] debug2: Testing job time limits and checkpoints
[2020-08-04T16:25:45.307] debug:  backfill: beginning
[2020-08-04T16:25:45.307] debug:  backfill: no jobs to backfill
[2020-08-04T16:25:51.937] debug2: Processing RPC: MESSAGE_NODE_REGISTRATION_STATUS from uid=0
[2020-08-04T16:25:51.937] Node devslurmvm02 rebooted 7 secs ago
[2020-08-04T16:25:51.937] node devslurmvm02 returned to service
[2020-08-04T16:25:51.937] debug2: _slurm_rpc_node_registration complete for devslurmvm02 usec=93
[2020-08-04T16:25:55.582] debug2: Processing RPC: MESSAGE_NODE_REGISTRATION_STATUS from uid=0
[2020-08-04T16:25:55.582] node devslurmvm01 returned to service
[2020-08-04T16:25:55.582] debug2: _slurm_rpc_node_registration complete for devslurmvm01 usec=61
[2020-08-04T16:26:07.086] debug2: Processing RPC: REQUEST_PARTITION_INFO uid=0
[2020-08-04T16:26:07.086] debug2: _slurm_rpc_dump_partitions, size=198 usec=55
[2020-08-04T16:26:15.236] debug2: Testing job time limits and checkpoints
[2020-08-04T16:26:15.236] debug2: Performing purge of old job records
[2020-08-04T16:26:15.236] debug:  sched: Running job scheduler
[2020-08-04T16:26:15.307] debug:  backfill: beginning
[2020-08-04T16:26:15.307] debug:  backfill: no jobs to backfill
[2020-08-04T16:26:21.410] debug2: Processing RPC: REQUEST_PARTITION_INFO uid=0
[2020-08-04T16:26:21.410] debug2: _slurm_rpc_dump_partitions, size=198 usec=45
[2020-08-04T16:26:45.266] debug2: Testing job time limits and checkpoints
[2020-08-04T16:27:15.298] debug2: Testing job time limits and checkpoints
[2020-08-04T16:27:15.298] debug2: Performing purge of old job records
[2020-08-04T16:27:15.298] debug:  sched: Running job scheduler
[2020-08-04T16:27:15.298] debug2: Performing full system state save
[2020-08-04T16:27:15.303] debug2: Sending tres '1=9,2=6001,3=0,4=3,5=9,6=0,7=0,8=0' for cluster


[root@dev-slurm01 ~]# sinfo -lN
Tue Aug 04 16:29:08 2020
NODELIST      NODES PARTITION       STATE CPUS    S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON               
devslurmvm01      1      dev*     drained 4       4:1:1   3000        0      1       v1 Reboot ASAP : reboot 
devslurmvm02      1      dev*     drained 4       4:1:1   3000        0      1       v2 Reboot ASAP : reboot
Comment 1 Jason Booth 2020-08-04 14:41:05 MDT 
Closing duplicate.

*** This ticket has been marked as a duplicate of ticket 9513 ***