Ticket 11492 - slurmctld[46191]: error: High latency for 1000 calls to gettimeofday(): 707 microseconds
Summary: slurmctld[46191]: error: High latency for 1000 calls to gettimeofday(): 707 m...
Status: RESOLVED INFOGIVEN
Alias: None
Product: Slurm
Classification: Unclassified
Component: slurmctld (show other tickets)
Version: 20.11.4
Hardware: Linux Linux
: 4 - Minor Issue
Assignee: Nate Rini
QA Contact:
URL:
Depends on:
Blocks:
 
Reported: 2021-04-29 10:38 MDT by Cineca HPC Systems
Modified: 2021-04-30 08:46 MDT (History)
0 users

See Also:
Site: Cineca
Slinky Site: ---
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
Google sites: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
NoveTech Sites: ---
Nvidia HWinf-CS Sites: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Tzag Elita Sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed:
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this ticket.
Description Cineca HPC Systems 2021-04-29 10:38:15 MDT 
Hi,

Debugging the problem reported in Bug 11487 we have observed on our systems the following error:

 slurmctld[46191]: error: High latency for 1000 calls to gettimeofday(): 707 microseconds

Although we have lived with it for quite a long time without particolar problems, we have observed that the reason for such message is the clocksource being set at 'hpet'.
Changing it to tsc with the command:

echo tsc > /sys/devices/system/clocksource/clocksource0/current_clocksource 

reduced the latency to less than 50 microseconds.
Do you think that using hpet with such a long latency might affect slurm?
Would you recommend to use tsc on linux x86_64 systems to keep the latency as low as possible?

Regards,
Marcello
Comment 1 Nate Rini 2021-04-29 11:39:42 MDT 
(In reply to Cineca HPC Systems from comment #0)
> Do you think that using hpet with such a long latency might affect slurm?
Yes, slurm makes many calls to gettimeofday() and time() which results in Slurm running slower overall. It can also cause Slurm to do odd things like mismark jobs timestamps and possibly falsely trigger timeout errors. I don't think 707us is bad enough to trigger these issues though.

> Would you recommend to use tsc on linux x86_64 systems to keep the latency
> as low as possible?
Yes, it is recommended (assuming that tsc works with your hardware). I would suggest testing it first before making a permanent change. The kernel allows it to be switched live.
Comment 2 Cineca HPC Systems 2021-04-30 07:43:19 MDT 
(In reply to Nate Rini from comment #1)
> (In reply to Cineca HPC Systems from comment #0)
> > Do you think that using hpet with such a long latency might affect slurm?
> Yes, slurm makes many calls to gettimeofday() and time() which results in
> Slurm running slower overall. It can also cause Slurm to do odd things like
> mismark jobs timestamps and possibly falsely trigger timeout errors. I don't
> think 707us is bad enough to trigger these issues though.
> 
> > Would you recommend to use tsc on linux x86_64 systems to keep the latency
> > as low as possible?
> Yes, it is recommended (assuming that tsc works with your hardware). I would
> suggest testing it first before making a permanent change. The kernel allows
> it to be switched live.

Dear Nate,

we have a few clusters based on different architecture but on x86_64 we have different configuration using either hpet or tsc with no problem until yesterday (except for the message). 
I guess we will switch to tsc where we have hpet, although I vaguely remember a discussion 3-4 years ago about the lower reliability of tsc due to the variable frequencies of modern CPUs, plus it seems tsc can be used in Spectre attacks. Obviously, I can't remember the outcome of the discussion :D , reading around it seems that tsc is actually more reliable than expected as recent CPU include a "constant_tsc" registry which is running at nominal frequency and therefore is unaffected by actual cpu frequency changes due to turbo or power states.
Anyway feel free to close the issue if you don't see anything else to add.

Best Regards,
Marcello
Comment 3 Nate Rini 2021-04-30 08:46:36 MDT 
(In reply to Cineca HPC Systems from comment #2)
> we have a few clusters based on different architecture but on x86_64 we have
> different configuration using either hpet or tsc with no problem until
> yesterday (except for the message). 
We added an explicit check for slow gettimeofday() in bug#4961 due to how difficult it was in diagnosing issues related to it.

> I guess we will switch to tsc where we have hpet, although I vaguely
> remember a discussion 3-4 years ago about the lower reliability of tsc due
> to the variable frequencies of modern CPUs, plus it seems tsc can be used in
> Spectre attacks. Obviously, I can't remember the outcome of the discussion
> :D , reading around it seems that tsc is actually more reliable than
> expected as recent CPU include a "constant_tsc" registry which is running at
> nominal frequency and therefore is unaffected by actual cpu frequency
> changes due to turbo or power states.

Yes, there can be issues with tsc but this should never be an issue on a modern HPC system:
> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_for_real_time/7/html/reference_guide/chap-timestamping
> http://oliveryang.net/2015/09/pitfalls-of-TSC-usage/

> Anyway feel free to close the issue if you don't see anything else to add.
Closing ticket per your response.