Linux Pipes Are Slow

One of my sideprojects is intended to address this: <a href="https:&#x2F;&#x2F;lwn.net&#x2F;Articles&#x2F;976836&#x2F;" rel="nofollow">https:&#x2F;&#x2F;lwn.net&#x2F;Articles&#x2F;976836&#x2F;</a><p>The idea is a syscall for getting a ringbuffer for any supported file descriptor, including pipes - and for pipes, if both ends support using the ringbuffer they&#x27;ll map the same ringbuffer: zero copy IO, potentially without calling into the kernel at all.<p>Would love to find collaborators for this one :)

&gt; I do not know why the JMP is not just a RET, however.<p>This is caused by the CONFIG_RETHUNK option. In the disassembly from objdump you are seeing the result of RET being replaced with JMP __x86_return_thunk.<p><a href="https:&#x2F;&#x2F;github.com&#x2F;torvalds&#x2F;linux&#x2F;blob&#x2F;v6.1&#x2F;arch&#x2F;x86&#x2F;include&#x2F;asm&#x2F;linkage.h#L23">https:&#x2F;&#x2F;github.com&#x2F;torvalds&#x2F;linux&#x2F;blob&#x2F;v6.1&#x2F;arch&#x2F;x86&#x2F;include...</a><p><a href="https:&#x2F;&#x2F;github.com&#x2F;torvalds&#x2F;linux&#x2F;blob&#x2F;v6.1&#x2F;arch&#x2F;x86&#x2F;lib&#x2F;retpoline.S#L121">https:&#x2F;&#x2F;github.com&#x2F;torvalds&#x2F;linux&#x2F;blob&#x2F;v6.1&#x2F;arch&#x2F;x86&#x2F;lib&#x2F;ret...</a><p>&gt; The NOP instructions at the beginning and at the end of the function allow ftrace to insert tracing instructions when needed.<p>These are from the ASM_CLAC and ASM_STAC macros, which make space for the CLAC and STAC instructions (both of them three bytes in length, same as the number of NOPs) to be filled in at runtime if X86_FEATURE_SMAP is detected.<p><a href="https:&#x2F;&#x2F;github.com&#x2F;torvalds&#x2F;linux&#x2F;blob&#x2F;v6.1&#x2F;arch&#x2F;x86&#x2F;include&#x2F;asm&#x2F;smap.h#L17-L26">https:&#x2F;&#x2F;github.com&#x2F;torvalds&#x2F;linux&#x2F;blob&#x2F;v6.1&#x2F;arch&#x2F;x86&#x2F;include...</a><p><a href="https:&#x2F;&#x2F;github.com&#x2F;torvalds&#x2F;linux&#x2F;blob&#x2F;v6.1&#x2F;arch&#x2F;x86&#x2F;include&#x2F;asm&#x2F;cpufeatures.h#L264">https:&#x2F;&#x2F;github.com&#x2F;torvalds&#x2F;linux&#x2F;blob&#x2F;v6.1&#x2F;arch&#x2F;x86&#x2F;include...</a><p><a href="https:&#x2F;&#x2F;github.com&#x2F;torvalds&#x2F;linux&#x2F;blob&#x2F;v6.1&#x2F;arch&#x2F;x86&#x2F;kernel&#x2F;alternative.c#L265">https:&#x2F;&#x2F;github.com&#x2F;torvalds&#x2F;linux&#x2F;blob&#x2F;v6.1&#x2F;arch&#x2F;x86&#x2F;kernel&#x2F;...</a>

Calling Linux pipes &quot;slow&quot; is like calling a Toyota Corolla &quot;slow&quot;. It&#x27;s fast enough for all but the most extreme use cases. Are you racing cars? In a sport where speed is more important than technique? Then get a faster car. Otherwise stick to the Corolla.

This is a side note to the main point being made, but on modern CPUs, &quot;rep movsb&quot; is just as fast as the fastest vectorized version, because the CPU knows to accelerate it. The name of the kernel function &quot;copy_user_enhanced_fast_string&quot; hints at this: the CPU features are ERMS (&quot;Enhanced Repeat Move String&quot;, which makes &quot;rep movsb&quot; faster for anything above a certain length threshold) and FSRM (&quot;Fast Short Repeat Move String&quot;, which makes &quot;rep movsb&quot; faster for shorter moves too).

Something I didn&#x27;t see mentioned in the article about AVX512, aside from the xsave&#x2F;xrstor overhead, is that AVX512 is power hungry and causes CPU frequency scaling. See [1], [2] for details and as an example of how nuanced it can get.<p>[1] <a href="https:&#x2F;&#x2F;www.intel.com&#x2F;content&#x2F;dam&#x2F;www&#x2F;central-libraries&#x2F;us&#x2F;en&#x2F;documents&#x2F;cryptography-processing-with-3rd-gen-intel-xeon-scalable-processors-19-may-2021.pdf" rel="nofollow">https:&#x2F;&#x2F;www.intel.com&#x2F;content&#x2F;dam&#x2F;www&#x2F;central-libraries&#x2F;us&#x2F;e...</a><p>[2] <a href="https:&#x2F;&#x2F;www.intel.com&#x2F;content&#x2F;www&#x2F;us&#x2F;en&#x2F;developer&#x2F;articles&#x2F;technical&#x2F;accelerating-x265-with-intel-advanced-vector-extensions-512-intel-avx-512.html" rel="nofollow">https:&#x2F;&#x2F;www.intel.com&#x2F;content&#x2F;www&#x2F;us&#x2F;en&#x2F;developer&#x2F;articles&#x2F;t...</a>

Just about every form of IPC is &quot;slow&quot;. You have decided to pay a performance cost for safety.

I am again getting the hug of death of Hacker News. The situation is better than the last time thanks to caching WordPress pages, but loading the page can still take a few seconds, so bear with me!

I didn&#x27;t quite grasp why the original splice has to be so slow. They pointed out what made it slower than vmsplice - in particular allocating buffers and using scalar instructions - but why is this necessary? Why couldn&#x27;t splice just be reimplemented as vmsplice? I&#x27;m sure there is a good reason, but I&#x27;ve missed it.

Be interesting to see a version using io_uring, which I think would let you pre-share buffers with the kernel avoiding some copies, and avoid syscall overhead (though the latter seems negligible here).

A bold claim for a blog that takes about 20 seconds to load.

Haha. When I read the title I smiled. Linux pipes slow? Moook.. Now try Cygwin pipes. Thats what I call slow!<p>Anyway, nice article, its good to know whats going on under the hood.

This is a really cool post and that is a massive amount of throughput.<p>In my experience in data engineering, it’s very unlikely you can exceed 500mb&#x2F;s throughput of your business logic as most libraries you’re using are not optimized to that degree (SIMD etc.). That being said I think it’s a good technique to try out.<p>I’m trying to think of other applications this could be useful for. Maybe video workflows?

&gt; I do not know why the JMP is not just a RET, however.<p>The jump seems generated by the expansion of the `ASM_CLAC` macro, which is supposed to change the EFLAGS register ([1], [2]). However in this case the expansion looks like it does nothing (maybe because of the target ?). I &#x27;d be interested to know more about that. Call to the wild.<p>[1]: <a href="https:&#x2F;&#x2F;github.com&#x2F;torvalds&#x2F;linux&#x2F;blob&#x2F;master&#x2F;arch&#x2F;x86&#x2F;include&#x2F;asm&#x2F;smap.h#L17">https:&#x2F;&#x2F;github.com&#x2F;torvalds&#x2F;linux&#x2F;blob&#x2F;master&#x2F;arch&#x2F;x86&#x2F;inclu...</a><p>[2]: <a href="https:&#x2F;&#x2F;stackoverflow.com&#x2F;a&#x2F;60579385" rel="nofollow">https:&#x2F;&#x2F;stackoverflow.com&#x2F;a&#x2F;60579385</a>

FUSE can be a bit trickier than a single queue of data chunks. Reads from &#x2F;dev&#x2F;fuse actually pick the right message to read based on priorities, and there&#x27;s cases where the message queue is meddled with to e.g. cancel requests before they&#x27;re even sent to userspace. If you naively switch it to eagerly putting messages into a userspace-visible ringbuffer, you might significantly change behavior in cases like interrupting slow operations. Imagine having to fulfill a ringbuf worth of requests to a misbehaving backend taking 5sec&#x2F;op, just to see the cancellations at the very end.

How do you gather profiling information for kernel function calls from a user program?

&gt; Although SSE2 is always available on x86-64, I also disabled the cpuid bit for SSE2 and SSE to see if it could nudge glibc into using scalar registers to copy data. I immediately got a kernel panic. Ah, well.<p>I think you need to recompile your compiler, or disable those explicitly via link &#x2F; cc flags. Compilers are fairly hard to get to coax &#x2F; dissuade SIMD instructions, IMHO.

I know pipes primarily from shell scripts. Are they being used in other contexts as extensively, too? Like C or Rust programs?

Someone tasted a bread thinking it is not sweet enough, which is fine. But calling the bread bland is funny because it does not mean to taste sweet.

Great post! I didn&#x27;t know about vmsplice(2). I&#x27;m glad to see a former ENSL student here as well!

Excellent article even if, to be honest, the title is clickbait.

I get PR_CONNECT_RESET_ERROR when trying to open the page

What is the library used to profile the program?

So is Python, but I&#x27;m still gonna use it lol

just never use pipes. they are some weird archaism that need to die :P<p>the only time ive used them is external constraints. they are just not useful.