Yes, this is why we (Netflix) default to tsc over the xen clocksource. I found the xen clocksource had become a problem a few years ago, quantified using flame graphs, and investigated using my own microbenchmark.<p>Summarized details here:<p><a href="https://www.slideshare.net/brendangregg/performance-tuning-ec2-instances/42" rel="nofollow">https://www.slideshare.net/brendangregg/performance-tuning-e...</a>
Another option is to reduce usage of gettimeofday() when possible. It is not always free.<p>Roughly 10 years ago, when I was the driver author for one of the first full-speed 10GbE NICs, we'd get complaints from customers that were sure our NIC could not do 10Gbs, as iperf showed it was limited to 3Gb/s or less. I would ask them to re-try with netperf, and they'd see full bandwidth. I eventually figured out that the complaints were coming from customers running distros without the vdso stuff, and/or running other OSes which (at the time) didn't support that (Mac OS, FreeBSD). It turns out that the difference was that iperf would call gettimeofday() around every socket write to measure bandwidth. But netperf would just issue gettimeofday calls at the start and the end of the benchmark, so iperf was effectively gettimeofday bound. Ugh.
The title is misleading. 77% slower sounds like the system calls take 1.77x the time on EC2. In fact, the results indicate that the normal calls are 77% faster - in other words, EC2 gettimeofday and clock_gettime calls take <i>nearly 4.5x longer</i> to run on EC2 than they do on ordinary systems.<p>This is a <i>big</i> speed hit. Some programs can use gettimeofday extremely frequently - for example, many programs call timing functions when logging, performing sleeps, or even constantly during computations (e.g. to implement a poor-man's computation timeout).<p>The article suggests changing the time source to tsc as a workaround, but also warns that it could cause unwanted backwards time warps - making it dangerous to use in production. I'd be curious to hear from those who are using it in production how they avoided the "time warp" issue.
I prefer the way Solaris solved this problem:<p>1) first, by eliminating the need for a context switch for libc calls such as gettimeofday(), gethrtime(), etc. (there is no public/supported interface on Solaris for syscalls, so libc would be used)<p>2) by providing additional, specific interfaces with certain guarantees:<p><a href="https://docs.oracle.com/cd/E53394_01/html/E54766/get-sec-fromepoch-3c.html" rel="nofollow">https://docs.oracle.com/cd/E53394_01/html/E54766/get-sec-fro...</a><p>This was accomplished by creating a shared page in which the time is updated in the kernel in a page that is created during system startup. At process exec time that page is mapped into every process address space.<p>Solaris' libc was of course updated to simply read directly from this memory page. Of course, this is more practical on Solaris because libc and the kernel are tightly integrated, and because system calls are not public interfaces, but this seems greatly preferable to the VDSO mechanism.
Author here, greetings. Anyone who finds this interesting may also enjoy our writeup describing every Linux system call method in detail [1].<p>[1]: <a href="https://blog.packagecloud.io/eng/2016/04/05/the-definitive-guide-to-linux-system-calls/" rel="nofollow">https://blog.packagecloud.io/eng/2016/04/05/the-definitive-g...</a>
For anyone looking at the mentions of KVM "under some circumstances" having the same issue and wondering how to avoid it with KVM: KVM appears to support fast vDSO-based time calls as long as:<p>- You have a stable hardware TSC (you can check this in /proc/cpuinfo on the host, but all reasonably recent hardware should support this).<p>- The host has the host-side bits of the KVM pvclock enabled.<p>As long as you meet those two conditions, KVM should support fast vDSO-based time calls.
So… it's not that the syscalls are slower, it's that the Linux-specific mechanism the Linux kernel uses to bypass having to actually perform these calls does not currently work on Xen (and thus EC2).
This was also presented at the last AWS re:Invent in December. See AWS EC2 Deep Dive: <a href="https://de.slideshare.net/mobile/AmazonWebServices/aws-reinvent-2016-deep-dive-on-amazon-ec2-instances-featuring-performance-optimization-best-practices-cmp301" rel="nofollow">https://de.slideshare.net/mobile/AmazonWebServices/aws-reinv...</a>
Interesting way to find out the version of the hypervisor kernel. If the gtod call returns faster than the direct syscall for it, then you know the kernel version is prior to that of the patch fixing the issue in xen.<p>I expect there are many such patches that you could use to narrow down the version range of the host kernel. Once you've that information, you may be in a better position to exploit it, knowing which bugs are and are not patched.
vDSO maintainer here.<p>There are patches floating around to support vDSO timing on Xen.<p>But isn't AWS moving away from Xen or are they just moving away from Xen PV?
Does anyone have any intuition around how this affects a variety of typical workflows? I imagine that these two syscalls are disproportionally likely to affect benchmarks more than real-world usage. How many times is this syscall happening on a system doing things like serving HTTP, or running batch jobs, or hosting a database, etc?
I wonder if they tried this:
<a href="https://blog.packagecloud.io/eng/2017/02/21/set-environment-variable-save-thousands-of-system-calls/" rel="nofollow">https://blog.packagecloud.io/eng/2017/02/21/set-environment-...</a>
Is this just an EC2 problem, or does it affect any Xen/KVM guest?<p>I ran the test program on a Hyper-V VM running CentOS 7 and got the same result: 100 calls to the gettimeofday syscall. Conversely, I tested a vSphere guest (also running CentOS 7), which didn't call gettimeofday at all.
Wasn't a workaround posted for this some time ago, that requires setting the TZ environment variable?<p><a href="https://news.ycombinator.com/item?id=13697555" rel="nofollow">https://news.ycombinator.com/item?id=13697555</a><p>It seems very closely related, unless I am mistaken.
How common are get time calls so that they would actually be an issue?<p>I've worked on quite a few systems and can't think of a time where an api for getting the time would have been called so much that it would affect performance?
OpenJDK has an open issue about this in their JVM: <a href="https://bugs.openjdk.java.net/browse/JDK-8165437" rel="nofollow">https://bugs.openjdk.java.net/browse/JDK-8165437</a>
> All programmers deploying software to production environments should regularly strace their applications in development mode and question all output they find.<p>Or, instead, you could just not do that. Then you could go back to being productive, instead of wasting time tracking down unstable small tweaks for edge cases that you can barely notice after looping the same syscall 5 million times in a row.<p>When will people learn not to micro-optimize?