Ticket 9798

Summary: We have more time than is possible for period during maintenance while upgrading slurm
Product: Slurm Reporter: john.blaas
Component: AccountingAssignee: Albert Gil <agilmoreno>
Status: RESOLVED DUPLICATE QA Contact:
Severity: 4 - Minor Issue    
Priority: --- CC: jonathon.anderson, shilton
Version: 20.02.4   
Hardware: Linux   
OS: Linux   
See Also: https://bugs.schedmd.com/show_bug.cgi?id=8555
Site: University of Colorado Slinky Site: ---
Alineos Sites: --- Atos/Eviden Sites: ---
Confidential Site: --- Coreweave sites: ---
Cray Sites: --- DS9 clusters: ---
Google sites: --- HPCnow Sites: ---
HPE Sites: --- IBM Sites: ---
NOAA SIte: --- NoveTech Sites: ---
Nvidia HWinf-CS Sites: --- OCF Sites: ---
Recursion Pharma Sites: --- SFW Sites: ---
SNIC sites: --- Tzag Elita Sites: ---
Linux Distro: --- Machine Name:
CLE Version: Version Fixed:
Target Release: --- DevPrio: ---
Emory-Cloud Sites: ---
Attachments: Slurmdbd log
slurm.conf

Description john.blaas 2020-09-10 10:34:02 MDT 
Created attachment 15842 [details]
Slurmdbd log

Hello,

Yesterday we had a maintenance where we upgraded slurm from 19.05 to 20.02-4. Once the maintenance started and we upgraded slurmdbd we started seeing error messages from our Summit cluster to the effect of "We have more time than is possible" but only for the hours in which we had the nodes down and partitions stopped.

This doesn't appear to be causing issues, aside from generating logs, but was wondering if you could confirm if there is a problem or whether these are innocuous messages.

Attached are the slurmdbd log and slurm.conf for the cluster in question.
Comment 1 john.blaas 2020-09-10 10:34:25 MDT 
Created attachment 15843 [details]
slurm.conf
Comment 3 Albert Gil 2020-09-14 00:48:21 MDT 
Hi John,

This bug looks like a possible duplicate of bug 6839, already fixed as part of the next major release 20.11.
Can you confirm if those down nodes were part of a reservation?
If they were, could you post the Flags of those reservations?

Thanks,
Albert
Comment 4 john.blaas 2020-09-14 12:19:07 MDT 
Albert,

The nodes were down as part of a reservation and the logs only indicate lost time while the maintenance was active.

flags=MAINT,PART_NODES is what was used for flags.

Thanks,
John
Comment 5 Albert Gil 2020-09-14 13:22:07 MDT 
(In reply to john.blaas from comment #4)
> The nodes were down as part of a reservation and the logs only indicate lost
> time while the maintenance was active.
> 
> flags=MAINT,PART_NODES is what was used for flags.

In this case I think that you are hitting the known bug 6839.
In bug 6839 comment 37 you can see that to fix the issue we needed to change other sensitive parts of the code, so we decided to add the fix only as part of the future major release 20.11.
You can see the main commit on:
- https://github.com/SchedMD/slurm/commit/71e7577efe6289a17f111c7cce2af03220343a82

(note that RESERVE_FLAG_MAINT is impacted)

If this is ok for you I'll close this bug as duplicate of 6839, and once the next major version is released, your problem should be gone.
If not, we should reopen it, of course.

Is that ok for you?

Thanks,
Albert
Comment 6 john.blaas 2020-09-14 17:36:44 MDT 
That sounds fine, we'll just reopen the case if it ends up causing issues.

Thanks,
John
Comment 7 Albert Gil 2020-09-15 01:44:23 MDT 
Closing as duplicate of bug 6839.

*** This ticket has been marked as a duplicate of ticket 6839 ***