> <i>In other words: your system supports calling the time system call via the Linux kernel’s vDSO to avoid the cost of switching to the kernel. But, as soon as your program calls time, it calls localtime immediately after, which invokes a system call anyway.</i><p>This reminds me of an article by Ted Unangst[1], in which he flattens the various libraries and abstractions to show how xterm (to cite one of many culprits) in one place is effectively doing:<p><pre><code> if (poll() || poll())
while (poll()) {
/* ... */
}
</code></pre>
In other words, if you don't know what your library/abstraction is doing, you can end up accidentally duplicating its work.<p>Reminds me of some aphorism, "Those who do not learn from history..." ;)<p>[1] <a href="http://www.tedunangst.com/flak/post/accidentally-nonblocking" rel="nofollow">http://www.tedunangst.com/flak/post/accidentally-nonblocking</a><p>discussed <a href="https://news.ycombinator.com/item?id=11847529" rel="nofollow">https://news.ycombinator.com/item?id=11847529</a>
System calls in Linux are really fast. So saving "thousands" of system calls when /etc/localtime is in cache doesn't actually save that much actual CPU time.<p>I ran an experiment where I timed the runtime of the sample program provided in the OP, except I changed the number of calls to localtime() from ten times to a million. I then timed the difference with and without export TZ=:/etc/localhost. The net savings was .6 seconds. So for a single call to localtime(3), the net savings is 0.6 microseconds.<p>That's non-zero, but it's likely in the noise compared to everything else that your program might be doing.
Good blog post explaining the behavior of glibc, I also saw this first hand when profiling Apache awhile back too:<p><a href="http://mail-archives.apache.org/mod_mbox/httpd-dev/201111.mbox/%3CCAMDeyhzRAZ4eyz%3D%2BstA%3DwoTibM-W6QL8TqT%2BaPio07UddCz7Tg%40mail.gmail.com%3E" rel="nofollow">http://mail-archives.apache.org/mod_mbox/httpd-dev/201111.mb...</a><p><a href="https://github.com/apache/httpd/blob/trunk/server/util_time.c#L310-L327" rel="nofollow">https://github.com/apache/httpd/blob/trunk/server/util_time....</a><p>The internals of glibc can often be pretty surprising sometimes, I'd really encourage people to go spelunking into the glibc source when they are profiling applications.
Please quantify the speedup (I've found this before, but it's never been a significant issue). Eliminating unnecessary work is great, but what are we really talking about here? Use a CPU flamegraph, Ctrl-F and search for stat functions. It'll quantify the total on the bottom right.
Honestly, the primary reason I support this is to get developers out of the habbit of demanding a localized server timezone. As an infra' person, I want system time in UTC. If developers get in the habbit of setting TZ, then I can have this!
Author of the post here: greetings.<p>If you enjoyed this post, you may also enjoy our deep dive explaining exactly how system calls work on Linux[1].<p>[1]: <a href="https://blog.packagecloud.io/eng/2016/04/05/the-definitive-guide-to-linux-system-calls/" rel="nofollow">https://blog.packagecloud.io/eng/2016/04/05/the-definitive-g...</a>
Is there a reason why the path to the timezone file is prefixed with a colon?<p>TZ=:/etc/localtime<p>I've set TZ sometimes without the colon and it seem to work. I did a quick online search and didn't find anything relevant.
Brendan Gregg wrote about this a few years ago [1].<p>My favorite part:<p>> WTF?? Why is ls(1) running stat() on /etc/localtime for every line of output?<p>[1] <a href="http://www.brendangregg.com/blog/2014-05-11/strace-wow-much-syscall.html" rel="nofollow">http://www.brendangregg.com/blog/2014-05-11/strace-wow-much-...</a>
What is missing in this post is:<p>- Why does glibc check /etc/localtime every time localtime is called? Wild guess: so that new values of /etc/localtime are picked at runtime without restarting programs.<p>- Corollary: why does glibc <i>not</i> check /etc/localtime every time localtime is called, when TZ is set to :/etc/localtime? Arguably the reason above should still apply when TZ is set to a file name, shouldn't it?
If this has a real-world/measurable/etc. impact why isn't this set by default? Are there potential side-effects? Is it set in some distros but not others?
This seems to be a simple RTFM issue to me: POSIX specifies that gmtime() uses UTC and localtime() uses current timezone. Using gmtime() would implement the desired behaviour without any need to hardcode environment variables.
Great post. I remember when vDSOs were added we noticed a nice speedup in our code. We tuned for realtime and a few microseconds here and there add up. Most importantly, less systems calls means more predictability.
This reminds me of a very similar behavior in Solaris over 20 years ago. Our C application was having odd performance problems on some client systems, and eventually we saw via truss that there were hundreds of fopen() calls every second to get the timezone. Setting the right environment variable solved the problem.
I really enjoy when people dig into things like this and report their findings. Having said that, I question the wisdom of "bothering" with this sort of thing. Everything you do that's non-standard or works against a system's default behavior incurs a cost. It's yet another thing you have to replicate when you migrate to a new version, change provisioning systems, etc.<p>And for what benefit? A few hundred syscalls per second? Linux syscalls are fast enough that something of that magnitude shouldn't matter much. Given that /etc/localtime will certainly be in cache with that frequency of access, a stat() should do little work in the kernel to return, so that won't be slow either.<p>It's good that they did some benchmarking to look at the differences, but this feels like a premature optimization to me. I can't imagine that this did anything but make their application a tiny fraction of a percent faster. Was it worth the time to dig into that for this increase? Was it worth the maintenance cost I mention in my first paragraph? I wouldn't think so.<p>I'm really trying not to take a crap on what they did; as I said, it's really cool to dig into these sorts of abstractions and find out where they're inefficient or leak (or just great as a learning exercise; we all depend on a mountain of code that most people don't understand at all). But, when looked at from a holistic systems approach, a grab bag of little "tweaks" like this can become harmful in the long run.
There's another easy way to avoid this: use localtime_r instead of localtime. From the glibc source:<p><pre><code> /* Update internal database according to current TZ setting.
POSIX.1 8.3.7.2 says that localtime_r is not required to set tzname.
This is a good idea since this allows at least a bit more parallelism. */
tzset_internal (tp == &_tmbuf && use_localtime, 1);
</code></pre>
mktime also does the tzset call every time, though:<p><pre><code> time_t
mktime (struct tm *tp)
{
#ifdef _LIBC
/* POSIX.1 8.1.1 requires that whenever mktime() is called, the
time zone names contained in the external variable 'tzname' shall
be set as if the tzset() function had been called. */
__tzset ();
#endif
</code></pre>
and I don't see any way around that other than setting TZ=: or some such.
BTW, packagecloud.io is the great hosting for RPM/DEB packages. We've been using it for the last couple years. GitHub + Travis CI + PackageCloud combination allows us build and publish packages for EVERY git commit in 30+ repositories targeting 15 different Linux distributions [1]. There is no more need to hire a special devops guy for that.<p>[1]: <a href="https://github.com/packpack/packpack#packpack" rel="nofollow">https://github.com/packpack/packpack#packpack</a>
Interesting article but I can't reproduce the behavior on Ubuntu 16.01 LTS. I don't have TZ set (or anything locale-related for that matter). Here are the library dependencies:<p><pre><code> $ ldd test
linux-vdso.so.1 => (0x00007ffd80baf000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8844bf7000)
/lib64/ld-linux-x86-64.so.2 (0x00007f8844fbc000)
</code></pre>
Any thoughts why the behavior would be different?
This reminds me of setting noatime for disk mounts (<a href="http://askubuntu.com/questions/2099/is-it-worth-to-tune-ext4-with-noatime" rel="nofollow">http://askubuntu.com/questions/2099/is-it-worth-to-tune-ext4...</a>)<p>Now I want to know the number of other configs to reduce the number of system calls. This all adds up to being significant the greater the number of hosts in your environment.
While trying to find the cause of slowness in Rails requests, I was running strace on an unicorn process when I encountered the same thing mentioned in the article.<p>Rails instrumentation code calls current time before and after any instrumentation block. So, when I looked at the trace there were a lot of `stat` calls coming for `/etc/localtime` and as stat is an IO operation, I thought I discovered the cause of slowness(which I attributed to high number of IO ops) but surprisingly when I saw the strace method summary; while the call count was high, the time taken by the calls in total was not significant(<1% if I remember correctly). So I decided to set TZ with the next AMI update 15 months back but forgot about it totally. I guess I should add it to my Trello list this time.<p>Also, I think he should have printed the aggregate summary of just CPU clock time(`-c`) as well as that is usually very low.
Really interesting - thanks for sharing the findings. I haven't seen it mentioned here, but for those of us using `timedatectl` via systemd, with the default setting of `UTC` are taking advantage[1] of the recommendation in the article.<p>[1] <a href="https://github.com/systemd/systemd/blob/master/src/timedate/timedatectl.c#L85" rel="nofollow">https://github.com/systemd/systemd/blob/master/src/timedate/...</a>
This reads to me like a glibc bug. Glibc should just be watching "/etc/localtime" for changes, rather than calling out to hundreds of times a second.
Side note - why do some sites completely hide information about who is behind them? I couldn't find a single thing about that on their blog or main site.
Im not an expert but the first thing that comes to mind is that
1) TFA does not quantify the performance gain in time
2) I wonder if environment variables like TZ are a security risk/vector in that these might facilitate attackers to stealthy skew/screw time within current user process... no root required.
Does anyone have any evidence of this actually having a resource usage impact on any common programs?<p>I see one reference to Apache below, but not whether it actually made a measurable difference.