A random thing I ran into with the defaults (Ubuntu Linux):<p>- net.ipv4.tcp_rmem ~ 6MB<p>- net.core.rmem_max ~ 1MB<p>So.. the tcp_rmem value overrides by default, meaning that the TCP receive window for a vanilla TCP socket actually goes up to 6MB if needed (in reality - 3MB because of the halving, but let's ignore that for now since it's a constant).<p>But if I "setsockopt SO_RCVBUF" in a user-space application, I'm actually capped at a maximum 1MB, even though I already have 6MB. If I try to <i>reduce it</i> from 6MB to e.g. 4MB, it will result in 1MB. This seems very strange. (Perhaps I'm holding it wrong?)<p>(Same applies to SO_SNDBUF/wmem...)<p>To me, it seems like Linux is confused about the precedence order of these options. Why not have core.rmem_max be larger and the authoritative directive? Is there some historical reason for this?
This is great, not just the parameters themselves but all the steps that a packet follows from the point it enters the NIC until it gets to userspace.<p>Just one thing to add regarding network performance: if you're working in a system with multiple CPUs (which is usually the case in big servers), check NUMA allocation. Sometimes the network card will be in one CPU while the application is executing on a different one, and that can affect performance too.
Just changing Linux's default congestion control (net.ipv4.tcp_congestion_control) to 'bbr' can make a _huge_ difference in some scenarios, I guess over distances with sporadic packet loss and jitter, and encapsulation.<p>Over the last year, I was troubleshooting issues with the following connection flow:<p>client host <-- HTTP --> reverse proxy host <-- HTTP over Wireguard --> service host<p>On average, I could not get better than 20% theoretical max throughput. Also, connections tended to slow to a crawl over time. I had hacky solutions like forcing connections to close frequently. Finally switching congestion control to 'bbr' gives close to theoretical max throughput and reliable connections.<p>I don't really understand enough about TCP to understand why it works. The change needed to be made on both sides of Wireguard.
Does performance tuning for Wi-Fi adapters matter?<p>On desktops, other than disabling features, can anything fix the problems with i210 and i225 ethernet? Those seem to be the two most common NICs nowadays.<p>I don't really understand why common networking hardware and drivers are so flawed. There is a lot of attention paid to RISC-V. How about start with a fully open and correct NIC? They'll shove it in there if it's cheaper than an i210. Or maybe that's impossible.
Great overview of the Linux network queues as provided in the Figure, should paste it on the wall somewhere.<p>Brendan's System Performance books provide nice coverage on Linux network performance and more [1]. It's already in the second edition, both are excellent books but the 2nd edition focuses mainly on Linux whereas the 1st edition also include Solaris.<p>There's also a more recent book on BPF Performance Tools by him [2].<p>[1] Systems Performance: Enterprise and the Cloud, 2nd Edition (2020)<p><a href="https://www.brendangregg.com/systems-performance-2nd-edition-book.html" rel="nofollow noreferrer">https://www.brendangregg.com/systems-performance-2nd-edition...</a><p>[2] BPF Performance Tools:<p><a href="https://www.brendangregg.com/bpf-performance-tools-book.html" rel="nofollow noreferrer">https://www.brendangregg.com/bpf-performance-tools-book.html</a>
I enjoyed skimming through the article. Very well researched and presented.<p>But can anybody help me out, who tunes linux network parameters on a regular basis?
This doc kinda needs to say "TCP" somewhere, as it's very focused on TCP concerns - which is useful, people are mostly using TCP. The default UDP tunings are awfully low and as such are notably missing.
Could anyone recommend a video or video series covering similar material?<p>There lots on networking in general, but I've had a hard time finding some on Linux specific implementation