SchedMD - Slurm Support – Full Text Ticket Listing
- Home
- | New
- | Browse
- | Search
- | [?]
- | Reports
- | Help
- | New Account
- | Log In
- | Forgot Password
| Summary: | error: We have more time than is possible | ||
|---|---|---|---|
| Product: | Slurm | Reporter: | surendra <surendra.sunkari> |
| Component: | Database | Assignee: | Gavin D. Howard <gavin> |
| Status: | RESOLVED DUPLICATE | QA Contact: | |
| Severity: | 3 - Medium Impact | ||
| Priority: | --- | ||
| Version: | 19.05.2 | ||
| Hardware: | Linux | ||
| OS: | Linux | ||
| Site: | NLR | Slinky Site: | --- |
| Alineos Sites: | --- | Atos/Eviden Sites: | --- |
| Confidential Site: | --- | Coreweave sites: | --- |
| Cray Sites: | --- | DS9 clusters: | --- |
| Google sites: | --- | HPCnow Sites: | --- |
| HPE Sites: | --- | IBM Sites: | --- |
| NOAA SIte: | --- | NoveTech Sites: | --- |
| Nvidia HWinf-CS Sites: | --- | OCF Sites: | --- |
| Recursion Pharma Sites: | --- | SFW Sites: | --- |
| SNIC sites: | --- | Tzag Elita Sites: | --- |
| Linux Distro: | --- | Machine Name: | |
| CLE Version: | Version Fixed: | ||
| Target Release: | --- | DevPrio: | --- |
| Emory-Cloud Sites: | --- | ||
Thank you for the report. To clarify: are you still seeing those log messages after clearing runaway jobs? I am still seeing the error messages. There is a known bug (Bug 6839) with reservation accounting that causes this error message. Of course, this only applies if you currently have reservations. Do you? Yes. we do have a system reservation in place now, as we are working on recovering the system from the power outage. Do you want to check on the logs after the system reservation is released and get back to you? Yes, please. Also, we are making progress on Bug 6839, so stay tuned. Just a reminder to send me the logs after the reservation is removed. If I haven't received logs in a week, I will need to close this ticket. We haven't seen that error messages on the logs after we reservation was cleared. you can close the request. Thank you for letting me know. *** This ticket has been marked as a duplicate of ticket 6839 *** |
We had a power outage and the machine running slurmctld/slurmdbd was shutdown abruptly. We are seeing these messages on the slurmdbd logs. [2019-11-18T13:24:05.538] error: We have more time than is possible (0+0+284212800)(284212800) > 283176000 for cluster eagle(78660) from 2019-11-18T02:00:00 - 2019-11-18T03:00:00 tres 1 [2019-11-18T13:24:05.598] error: We have more time than is possible (0+0+284212800)(284212800) > 283176000 for cluster eagle(78660) from 2019-11-18T03:00:00 - 2019-11-18T04:00:00 tres 1 [2019-11-18T13:24:05.658] error: We have more time than is possible (0+0+284212800)(284212800) > 283176000 for cluster eagle(78660) from 2019-11-18T04:00:00 - 2019-11-18T05:00:00 tres 1 [2019-11-18T13:24:05.718] error: We have more time than is possible (0+0+284212800)(284212800) > 283176000 for cluster eagle(78660) from 2019-11-18T05:00:00 - 2019-11-18T06:00:00 tres 1 [2019-11-18T13:24:05.783] error: We have more time than is possible (0+0+284212800)(284212800) > 283176000 for cluster eagle(78660) from 2019-11-18T06:00:00 - 2019-11-18T07:00:00 tres 1 [2019-11-18T13:24:05.849] error: We have more time than is possible (0+0+284212800)(284212800) > 283176000 for cluster eagle(78660) from 2019-11-18T07:00:00 - 2019-11-18T08:00:00 tres 1 [2019-11-18T13:24:05.914] error: We have more time than is possible (0+0+284212800)(284212800) > 283176000 for cluster eagle(78660) from 2019-11-18T08:00:00 - 2019-11-18T09:00:00 tres 1 [2019-11-18T13:24:05.974] error: We have more time than is possible (0+0+285249312)(285249312) > 283176000 for cluster eagle(78660) from 2019-11-18T09:00:00 - 2019-11-18T10:00:00 tres 1 [2019-11-18T13:24:06.034] error: We have more time than is possible (0+0+285249600)(285249600) > 283176000 for cluster eagle(78660) from 2019-11-18T10:00:00 - 2019-11-18T11:00:00 tres 1 [2019-11-18T13:24:06.094] error: We have more time than is possible (0+0+285249600)(285249600) > 283176000 for cluster eagle(78660) from 2019-11-18T11:00:00 - 2019-11-18T12:00:00 tres 1 [2019-11-18T13:24:06.154] error: We have more time than is possible (0+0+285249600)(285249600) > 283176000 for cluster eagle(78660) from 2019-11-18T12:00:00 - 2019-11-18T13:00:00 tres 1 I cleared the orphaned jobs. [root@emgmt1 ~]# sacctmgr show runaway NOTE: Runaway jobs are jobs that don't exist in the controller but have a start time and no end time in the database ID Name Partition Cluster State TimeSubmit TimeStart TimeEnd ------------ ---------- ---------- ---------- ---------- ------------------- ------------------- ------------------- 520159 ddn_ior_s+ ddn eagle FAILED 2018-12-17T12:41:32 2018-12-17T12:41:32 Unknown 521837 sh short eagle FAILED 2018-12-18T16:48:22 2018-12-18T16:48:22 Unknown Would you like to fix these runaway jobs? (This will set the end time for each job to the latest out of the start, eligible, or submit times, and set the state to completed. Once corrected, this will trigger the rollup to reroll usage from before the earliest submit time of all the runaway jobs.) (You have 30 seconds to decide) (N/y): y [root@emgmt1 ~]# sacctmgr show runaway Runaway Jobs: No runaway jobs found on cluster eagle