The call shifts the loop from being limited at the backend level (mostly uops not being retired by having to wait for memory) to being limited at the frontend level. The core event to look for here is `IDQ_UOPS_NOT_DELIVERED.CORE`, which tells us whether the frontend is delivering the full 4 uops per cycle it is able to. In the tight loop this is almost always the case, whereas in the call loop this is rarely the case.<p>CALL, RET, and JNE all share the same execution port (6 in Skylake), so it seems plausible that the added pressure in this port prevents speculative execution from continuing with the loop at the same rate as the tight loop. If you look at the execution port breakdowns of each loop, port 6 dominates in the call loop, whereas the tight loop is bottlenecked at port 4 (the port where stores go).<p>By delivering fewer uops per cycle, the pressure on the backend is eased. But this is a delicate balance. If you add another call, the loop becomes much slower than the tight loop.<p>You can get a similar effect by replacing `__asm__("call foo")` with<p><pre><code> __asm__("jmp 1f\n1:\n");
__asm__("jmp 1f\n1:\n");
</code></pre>
which consumes the same amount of port 6.
Disclaimer: I am not an expert and have not measured. This is armchair theory. But, I would argue two things.<p>First, the former appears to have at least one unaligned arithmetic:<p>> <i>400538: mov 0x200b01(%rip),%rdx # 601040 <counter></i><p>...while the latter's equivalent instruction is 4-byte aligned:<p>> <i>40057d: mov 0x200abc(%rip),%rdx # 601040 <counter></i><p>So, I would argue that's the biggest source of _speedup_ in the second case. However, I'm really interested in whether that's true since I don't see a memory fence; so the memory should be in L0 cache for both cases; I have trouble believing that an unaligned access can be so much slower with the data in cache.<p>As for the `callq` to `repz retq`, I would venture a guess that the CPU's able to identify that there are no data dependencies there and the data's never even stored; I'd argue that it probably never even gets executed because the instruction should fit in instruction cache and branch prediction cache and all. Arguably. Like I said, I'm not an expert.<p>I'd say run it through Intel's code analyzer tool.<p><a href="https://software.intel.com/en-us/articles/intel-architecture-code-analyzer" rel="nofollow">https://software.intel.com/en-us/articles/intel-architecture...</a><p>Tangential video worth watching:<p><a href="https://www.youtube.com/watch?v=2EWejmkKlxs&feature=youtu.be&t=2466" rel="nofollow">https://www.youtube.com/watch?v=2EWejmkKlxs&feature=youtu.be...</a><p>Edit: actually, thinking about it, it's not unaligned <i>access</i>, it's unaligned <i>math</i>. I don't think that should affect performance at all? Fun.
Educated guess: The processor is trying to prefetch instructions. This loop is much tighter than most code that would typically be written, so an incrementing loop causes a branch misprediction. The processor is still loading instructions, so when it goes to find out what to do next, it takes a cache miss and burns some time trying to figure out its next instruction. However, a function call is very slow, however (even to a nop function) and it could delay the processor just long enough for the prefetch to complete.