Ticket 12705

Hi Aditi,

(In reply to Aditi Gaur from comment #0)
> We run slurm controller on kubernetes in a containerized environment. As a
> result the time zone set inside the container is in UTC. We cannot change
> this as all other containers (not created by us, or from third party) are
> also in UTC so the preference is to keep things consistent.
What is preventing you from changing the timezone of the containers? Is there not a way to override the timezone?

> However user environment is set by users where they can change the time zone
> tm_isdstto their preference by setting the TZ environment parameter. In our case
> most people have it in pacific. But when they submit an scrontab job to run
> at say 5 am every day scrontab only considers that as 5 AM UTC owing to
> slurmctld time zone inside the container.
Yes, it looks like scrontab does not do any sort of timezone conversion for any of the entries. All times are assumed to be in the timezone of the controller. This is a current limitation that we should document.

> We would prefer if that time zone be converted to UTC instead of defaulting
> to UTC and honor the user's time zone preference. Otherwise this will
> involve educating users what time zone slurmctld is running in which in this
> case is an unnecessary detail. In general this can be useful as well to
> allow users to work in their own time zones.
> 
> 1) Is there a way this is already supported and or if not- do you have a
> workaround?
Unfortunately, I'm not aware of an easy workaround other than changing the controller to match the timezone of the users or to educate the users on this limitation.

> 2) Can this be supported?
There may be a way to support automatically converting a cronjob from the user's timezone to the controller's timezone, but that would be an enhancement request that we would need to evaluate further. Daylight savings and other timezone issues make this a non-trivial fix.

Thanks!
-Michael

Hi Michael,

We can certainly change the time zone in the container but the problem is many other containers are also run for system management and its not feasible to change all time zones in a manner consistent with slurm. For logging correlation and troubleshooting we prefer all containers to log in 1 time zone which in this case happens to be UTC (all containers generally come with that pre-set).

In a distributed system it is reasonable to expect that time zones be converted to a consistent format before being processed. I certainly understand that the change is non-trivial but at the same time I think this more of a bug than an enhancement. User's shouldnt have to know the time zones in advance- and we have an international user base which makes this more impactful. We are really hoping to use scrontab as being our crontab solution for perlmutter and this is blocker at the moment for us.

My guess is this issue is not just limited to scrontab but any timestamps processed by slurmctld from a batch script. I think its reasonable to expect that slurm controller should be able to convert all timestamps before processing them.

In addition to Aditi's excellent points I'd just add that we are not the only consumers of the logs from the vast array of containers on the system; the vendor also gathers and uses these for their support purposes and so it's important that our Slurm containers use the same UTC time zone as all other on-system logs (whether from containers, kubernetes nodes or compute nodes).

> In a distributed system it is reasonable to expect that time zones be
> converted to a consistent format before being processed. I certainly
> understand that the change is non-trivial but at the same time I think this
> more of a bug than an enhancement. User's shouldnt have to know the time
> zones in advance- and we have an international user base which makes this
> more impactful. We are really hoping to use scrontab as being our crontab
> solution for perlmutter and this is blocker at the moment for us.

There's no such thing as "convert[ing] to a consistent format" where timezones are concerned. Daylight savings time and similar issues mean there is no single offset to UTC (or, worse yet, to another local timezone) you can rely on as a simplification, and the calculations would need to be handled in the submission timezone to do what you're requesting.

The only way to support this request would be to (a) capture the submission timezone from the user's environment, and then (b) use that to modify the next-eligible-time calculation.

(a) is simple enough, although would require RPC changes to support

(b) is actually quite difficult due to how time is handled through libc

The system call we rely on for these calculations - mktime() - does not have a variant that will allow it to calculate within a different timezone.

Instead, it relies on the process's current timezone, set through tzset(). But this is a _process wide_ setting. There's no way to change it just for the current thread. This means that, if we were to try to use tzset() to change this to allow the timezone to vary for each entry and enable mktime() to run the calculation in the desired timezone, that _all other threads would need to stop processing_, otherwise they may, e.g., emit log info with the incorrect timezone.

That's obviously not feasible. All other approaches require either re-implementing a lot of complex time and date math within Slurm, or splitting off a separate process (which could manipulate tzset() safely) to handle the next-eligible calculation. Neither of which are options I care to explore at the moment.

- Tim

P.S. this is not just a limitation for scrontab, but of cron itself: 

LIMITATIONS
       The  cron  daemon  runs with a defined timezone.  It currently does not
       support per-user timezones.  All the tasks: system's and user's will be
       run  based on the configured timezone.  Even if a user specifies the TZ
       environment variable in his crontab this will affect only the  commands
       executed  in  the crontab, not the execution of the crontab tasks them‐
       selves.

Hi Tim,

Do you have to send time zone information in the RPC? I was hoping this could be a client side improvement. For example scrontab time format could provide a time zone specification like PDT or PST. And then convert that to slurmctld's time before sending the rpc. You perhaps need to add an additional parameter to slurm.conf to specify slurmctld time zone but I think we certainly can consider a simpler approach here and not modify slurmctld at all.

In fact slurmctld's own time format is not a problem here at all since when we query slurm for any time related data it gets converted to the right time zone based on the environment. Its only the time that gets sent to the controller for scrontab and some sbatch options.

I understand that this is not a crontab feature but scrontab was meant to be an improvement over crontab for precisely that reason :)

(In reply to Aditi Gaur from comment #6)
> Hi Tim,
> 
> Do you have to send time zone information in the RPC? I was hoping this
> could be a client side improvement. For example scrontab time format could
> provide a time zone specification like PDT or PST. And then convert that to
> slurmctld's time before sending the rpc. You perhaps need to add an
> additional parameter to slurm.conf to specify slurmctld time zone but I
> think we certainly can consider a simpler approach here and not modify
> slurmctld at all.

I should have mentioned this, but you can already change the slurmctld's timezone just by setting the TZ environment variable appropriately. I do not see the need for a configuration setting for that at this time.

> In fact slurmctld's own time format is not a problem here at all since when
> we query slurm for any time related data it gets converted to the right time
> zone based on the environment. Its only the time that gets sent to the
> controller for scrontab and some sbatch options.
> 
> I understand that this is not a crontab feature but scrontab was meant to be
> an improvement over crontab for precisely that reason :)

The issue is that it has to be left up to the slurmctld to calculate the next valid repetition for a given crontab entry each time it needs to repeat. There is no way to pre-compute these client-side - it has to take the current time, then work out the next valid occurrence.

There's no way to handle an offset client side as you suggest: a crontab entry requesting the next repetition happen at 8:00 in the PST8PDT timezone does _not_ make sense without knowing what day it is.

- Tim

Hi Tim,

It seems CRON_TZ is supported in SLES. I am looking at man 5 crontab:

```
The  CRON_TZ  variable specifies the time zone specific for the cron table.  The user should enter a time according to the specified time zone into the table.  The time used for writing into a log
       file is taken from the local time zone, where the daemon is running.
```

```
#
       CRON_TZ=Japan
       # run five minutes after midnight, every day
       5 0 * * *       $HOME/bin/daily.job >> $HOME/tmp/out 2>&1
       # run at 2:15pm on the first of every month -- output mailed to paul
       15 14 1 * *     $HOME/bin/monthly
       # run at 10 pm on weekdays, annoy Joe
       0 22 * * 1-5    mail -s "It's 10pm" joe%Joe,%%Where are your kids?%
       23 0-23/2 * * * echo "run 23 minutes after midn, 2am, 4am ..., everyday"
       5 4 * * sun     echo "run at 5 after 4 every sunday"
```

Aditi -

Is there anything further I can help clarify here? I know we've been back and forth with this on Slack for a bit, but at the moment I still don't have anything I can propose to address this cleanly.

- Tim

Hi Tim

Yeah we are looking at accomplishing this via job submit filter if it is at all possible. I haven't had the time yet to dive deep into it- but what do you think of that idea? I am just looking for a suggestion here, but as far as this bug is concerned, I guess its fine to close it since we are clear that enhancement to the scheduler is not feasible.

To elaborate- submit filter does receive user's env- so we can have them set a CRON_TZ type param that details the desired TZ. Then we adjust the submit time from UTC to that TZ-- we can have hardcoded offsets to UTC-- and some manual date related constraints around daylight savings since US/Pacific moves around. This sounds not that great on paper- but i thought maybe its worth giving it a shot.

We can even have users define CRON_TZ specifically as PDT or PST which may make things easier.

(In reply to Aditi Gaur from comment #12)
> To elaborate- submit filter does receive user's env- so we can have them set
> a CRON_TZ type param that details the desired TZ. Then we adjust the submit
> time from UTC to that TZ-- we can have hardcoded offsets to UTC-- and some
> manual date related constraints around daylight savings since US/Pacific
> moves around. This sounds not that great on paper- but i thought maybe its
> worth giving it a shot.
> 
> We can even have users define CRON_TZ specifically as PDT or PST which may
> make things easier.

I wouldn't recommend this, although it could work in principle.

IMO, I think any half-functioning solution here is worse than the alternatives of either (a) explaining that the cron jobs are always UTC and letting the user decide how to handle that, or (b) changing the slurmctld to run in US/Pacific.

While you could mangle the cron spec (which isn't exposed to Lua at the moment) to adjust the hours by some fixed offset, this would not move the special values of @monthly, @weekly, and @daily/@midnight around which may lead to further confusion. And you're still stuck with not following the local timezone conventions around daylight savings time.

One suggestion - can you survey the existing users to get a feel for how prevalent these various patterns are? My naive assumption is that the majority of the entries are probably @hourly or @daily, or similar patterns, and for those the timezone really doesn't matter much. (Aside from what part of the day @daily runs in.)

So quick survey of all the cron jobs on cori (which at the moment isn't using scrontab) is that most people are specifying times exactly. I couldnt find any instance of "yearly", "hourly", "weekly" or "daily" and SLES crontab does support those nicknames though it does not seem to have a "midnight".


On perlmutter- only 2 staff folks are using cron for now- and they dont have any of the shortcuts.

I found this, I thought i ll put this here: 
https://unicode-org.github.io/icu/userguide/datetime/  
in case this ever becomes appealing.

ICU ** claims** that they provide timezone conversions based on local daylight conventions. But I think it only works in C++. I am not seeing a C implementation. 

I thought submit filter route would be easier but I hadn't factored in that cron spec isnt going into submit filter which obviously complicates things. And you are right that we might have to sacrifice the "weekly" and "monthly" shortcuts though an argument can be made that atleast at NERSC no one is using it.

I suppose I dont really have any other suggestions.