Created attachment 37873 [details]
slurmd log file

Hello Ole,

Thanks for the detailed writeup. I'm going to replicate this in my environment after carefully reviewing it, but it does look like slurmd does not honor the TTL field in the SRV record.

I'll work on verifying the problem and proposing a patch if confirmed. Just to make sure, this is not causing any pressing issue on your end?

Kind regards,
Daniel

Hi Daniel,

Thanks a lot for your quick response!

(In reply to Daniel Armengod from comment #2)
> Thanks for the detailed writeup. I'm going to replicate this in my
> environment after carefully reviewing it, but it does look like slurmd does
> not honor the TTL field in the SRV record.

Thanks for sharing offhand my view of the issue, let's see what comes up after your analysis.

> I'll work on verifying the problem and proposing a patch if confirmed. Just
> to make sure, this is not causing any pressing issue on your end?

That's correct, we made the workaround of quickly restarting all slurmd's after changing the DNS SRV record a few weeks ago, and this resolved the pressing issue. 

Other Slurm users may be surprised in the future if they encounter the same issue after updating the DNS SRV record, so it would be good to get a fix or a documentation update.

Best regards,
Ole

I have thought that the desired response of slurmd when it discovers that the TTL of the DNS SRV record has expired, is that it should create a new slurmd process, just like what happens with an "scontrol reconfigure" (from 23.11):

              Instruct all slurmctld and slurmd daemons to re-read the config‐
              uration file.  This mechanism can be used to  modify  configura‐
              tion  parameters  set in slurm.conf(5) without interrupting run‐
              ning jobs.  Starting in 23.11, this command operates by creating
              new  processes  for the daemons, then passing control to the new
              processes when or if they start up successfully. This allows  it
              to gracefully catch configuration problems and keep running with
              the previous configuration if there is a problem. 

Probably you'll agree on this?  BTW, have you reproduced the issue in your own environment?

Thanks,
Ole

(In reply to Ole.H.Nielsen@fysik.dtu.dk from comment #6)
> I have thought that the desired response of slurmd when it discovers that
> the TTL of the DNS SRV record has expired, is that it should create a new
> slurmd process, just like what happens with an "scontrol reconfigure" (from
> 23.11):

Actually, I meant to say that if the DNS SRV record has *changed*, then slurmd should be restarted.  It's only after the TTL has expired that slurmd is going to discover if it has a changed value.

/Ole

Hi Ole,

Just getting a chance to put down in writing what we had the opportunity to discuss in-person at SLUG'24.

Regardles of whether configless slurmd discovers its ctld through --conf-server or DNS SRV, fetching the config is a one-time operation. The config's "freshness" is not associated in any way with the DNS's record TTL field; this is akin to fetching a generic resource over the net.

I'm going to propose a patch to slightly alter the wording in the configless documentation to emphasize the fact that discovery is a one-time operation performed at startup and that the daemon runs normally afterwards.

Kind regards,
Daniel

Hi Daniel,

It was nice to meet you at the SLUG'24 meeting last week.  Thanks for taking the time to discuss this case with me.

I appreciate the clarification that the DNS SRV record's TTL value isn't used for anything at all by slurmd after starting up - this was unexpected to me, given the normal semantics of DNS TTL values!

(In reply to Daniel Armengod from comment #9)
> Just getting a chance to put down in writing what we had the opportunity to
> discuss in-person at SLUG'24.
> 
> Regardles of whether configless slurmd discovers its ctld through
> --conf-server or DNS SRV, fetching the config is a one-time operation. The
> config's "freshness" is not associated in any way with the DNS's record TTL
> field; this is akin to fetching a generic resource over the net.
> 
> I'm going to propose a patch to slightly alter the wording in the configless
> documentation to emphasize the fact that discovery is a one-time operation
> performed at startup and that the daemon runs normally afterwards.

It will be good to get a clarification in the configless documentation.

Conclusion: All slurmd's have to be restarted if the DNS SRV record is changed in a configless setup, is this correct?

I have a followup question:  If a site doesn't use configless, the slurmd reads slurm.conf from some file at startup and caches the value of SlurmctldHost.  What will happen when SlurmctldHost is changed in the slurm.conf file, similar to the present migration scenario?

My expectation is that slurmd will ignore the updated slurm.conf file until slurmd is explicitly restarted with "systemctl restart slurmd".  I don't know if a "scontrol reconfig" command executed on the new SlurmctldHost will be rejected or ignored by slurmd's?

It would be good to have a documentation of migrating SlurmctldHost which works both with the configless scenario and with a local slurm.conf file.

Thanks,
Ole

It would be good to

Hi Ole,

>It was nice to meet you at the SLUG'24 meeting last week.

Same! :)

>I have a followup question:  If a site doesn't use configless, the slurmd reads slurm.conf from some file at startup and caches the value of SlurmctldHost.  What will happen when SlurmctldHost is changed in the slurm.conf file, similar to the present migration scenario?

"Hot"-moving the slurmctld is not _really_ a common operation. As you surmise the extant slurmds will fail to contact the slurmctl and become stuck until timeouts happen. They *will* acknowledge and execute a reconfig RPC from the new ctld, but not until all running operations finish, which due to the fact that they cannot contact the slurmctld, will take a long time.

The recommended plan of action is to shut down the ctld, update slurm.conf, bring it up at a new location, and then restart all slurmds. It's cleaner that way.

>It would be good to have a documentation of migrating SlurmctldHost which works both with the configless scenario and with a local slurm.conf file.

A documentation patch is in the works. I originally thought to clarify only the SRV TTL issue, but I'll keep this in mind.

Hi Daniel,

(In reply to Daniel Armengod from comment #11)
> >I have a followup question:  If a site doesn't use configless, the slurmd reads slurm.conf from some file at startup and caches the value of SlurmctldHost.  What will happen when SlurmctldHost is changed in the slurm.conf file, similar to the present migration scenario?
> 
> "Hot"-moving the slurmctld is not _really_ a common operation. As you
> surmise the extant slurmds will fail to contact the slurmctl and become
> stuck until timeouts happen. They *will* acknowledge and execute a reconfig
> RPC from the new ctld, but not until all running operations finish, which
> due to the fact that they cannot contact the slurmctld, will take a long
> time.

I agree that moving the slurmctl to a new server is a rare operation, but we need to be able to do so reliably and without losing running jobs (which became possible with 23.11) when we have to move the slurmctld to a new OS or a new hardware.

Are you saying that slurmd would be blocking trying to contact the old SlurmctldHost until timeout, and only then perform the reconfig operation from the new SlurmctldHost?

> The recommended plan of action is to shut down the ctld, update slurm.conf,
> bring it up at a new location, and then restart all slurmds. It's cleaner
> that way.

Yes, I agree that this is the clean and reliable way!

I have now updated my Slurm Wiki page on this topic: https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_installation/#migrate-the-slurmctld-service-to-another-server  Could you perhaps look at the page quickly to see if I make any incorrect statements?

> >It would be good to have a documentation of migrating SlurmctldHost which works both with the configless scenario and with a local slurm.conf file.
> 
> A documentation patch is in the works. I originally thought to clarify only
> the SRV TTL issue, but I'll keep this in mind.

Thanks a lot for improving the documentation!  I'm sure it will help other users as well.

Please close this case now.

Best regards,
Ole

Ole,

Daniel brought this to my attention. I just want to clarify the role of our support 
folks when dealing with external documentation requests like this. We are going to 
have to decline your request to review the wiki link.

I would rather see changes and improvements in the official documentation. Request 
like this can give a false impression that SchedMD endorses what is found on your 
wiki and this is something we want to avoid. I do understand that we have in the 
past done this on occasion, but it is something we are trying to refrain from doing. 

I would ask that you keep your questions in the ticket and directed at the issue you 
are trying to solve.

> I have now updated my Slurm Wiki page on this topic: 
> https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_installation/#migrate-the-slurmctld-service-to-another-server  
> Could you perhaps look at the page quickly to see if I make any incorrect 
> statements?


Regarding you other questions: I will have Daniel answers those in the next update 
to this issue.

I'm out of the office, back on September 26.
Jeg er ikke på kontoret, tilbage igen 26. september.

Best regards / Venlig hilsen,
Ole Holm Nielsen

Hello Ole,

>Are you saying that slurmd would be blocking trying to contact the old SlurmctldHost until timeout, and only then perform the reconfig operation from the new SlurmctldHost?

Yes. As part of the reconfigure process slurmd must join all extant RPC handler threads. Threads that need to communicate with the controller will hang until timeout, at which point reconfigure will proceed.

Kind regards,
Daniel

Hi Jason,

Thanks for the note, and I fully agree with you that my Wiki is exclusively my own responsibility and that SchedMD should not comment upon it.

Best regards,
Ole

(In reply to Jason Booth from comment #13)
> Daniel brought this to my attention. I just want to clarify the role of our
> support 
> folks when dealing with external documentation requests like this. We are
> going to 
> have to decline your request to review the wiki link.
> 
> I would rather see changes and improvements in the official documentation.
> Request 
> like this can give a false impression that SchedMD endorses what is found on
> your 
> wiki and this is something we want to avoid. I do understand that we have in
> the 
> past done this on occasion, but it is something we are trying to refrain
> from doing.

Hi Daniel,

(In reply to Daniel Armengod from comment #16)
> >Are you saying that slurmd would be blocking trying to contact the old SlurmctldHost until timeout, and only then perform the reconfig operation from the new SlurmctldHost?
> 
> Yes. As part of the reconfigure process slurmd must join all extant RPC
> handler threads. Threads that need to communicate with the controller will
> hang until timeout, at which point reconfigure will proceed.

Thanks a lot for the clarification.  It seems that restart of all slurmd's is the correct approach when migrating the slurmctld.

Best regards,
Ole

Hello Ole,

Thank you very much for reporting this issue. The doc patch will land in 24.05.4.

If there's nothing else, I'm closing this ticket.

-Daniel