Ticket 11547 - Seeing "Could not open job state file" message, warns "Jobs may be lost!"
Summary: Seeing "Could not open job state file" message, warns "Jobs may be lost!"
Status: RESOLVED INFOGIVEN
Alias: None
Product: Slurm
Classification: Unclassified
Component: slurmctld (show other tickets)
Version: 20.11.5
Hardware: Linux Linux
: 4 - Minor Issue
Assignee: Jason Booth
QA Contact:
URL:
Depends on:
Blocks:
 
Reported: 2021-05-05 13:45 MDT by Will Dennis
Modified: 2021-05-05 20:12 MDT (History)
0 users

See Also:
Site: NEC Labs
Slinky Site: ---
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
Google sites: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
NoveTech Sites: ---
Nvidia HWinf-CS Sites: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Tzag Elita Sites: ---
Linux Distro: Ubuntu
Machine Name: ma-slurm-ctlr
CLE Version:
Version Fixed:
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this ticket.
Description Will Dennis 2021-05-05 13:45:36 MDT 
Hello,

When shutting down a 1st run of slurmctld (done via "slurmctld -Dvvvv"), I am seeing this in the log output:

slurmctld: Terminate signal (SIGINT or SIGTERM) received
slurmctld: debug:  sched: slurmctld terminating
slurmctld: debug3: _slurmctld_rpc_mgr shutting down
slurmctld: Saving all slurm state
slurmctld: debug:  create_mmap_buf: Failed to open file `/var/lib/slurm/slurmctld/job_state`, No such file or directory
slurmctld: error: Could not open job state file /var/lib/slurm/slurmctld/job_state: No such file or directory
slurmctld: error: NOTE: Trying backup state save file. Jobs may be lost!
slurmctld: debug:  create_mmap_buf: Failed to open file `/var/lib/slurm/slurmctld/job_state.old`, No such file or directory
slurmctld: No job state file (/var/lib/slurm/slurmctld/job_state.old) found
slurmctld: debug3: Writing job id 0 to header record of job_state file
slurmctld: debug3: _slurmctld_background shutting down

Shall I be concerned about this? If so, what to do to fix?
Comment 1 Jason Booth 2021-05-05 15:19:16 MDT 
Will - if this is the first time the scheduler has started and shutdown then what you are seeing is normal. Slurm will write out state information to the StateSaveLocation on shutdown. This includes information about jobs, partitions, nodes, associations, clustername, federation, database messages, config state, tres, qos, priority, reservations and triggers.

Since this is the first time the cluster has started it will not contain the state information for the cluster until the first shutdown. At this point, it will write this information out to the StateSaveLocation.
Comment 2 Will Dennis 2021-05-05 20:12:19 MDT 
It is the first time, but since this message is new to me, wanted to check it out. You may go ahead and close, thanks!