The Case of the 50ms request

Pretty fun puzzle, but this kind of debugging is alien to me. Without going into too much detail, it&#x27;s pretty easy to see something suspect is going on, and this hunch can be easily tested. This debugging tour takes you to strace, wireshark, all sorts of other low level debugging techniques, when really all you had to do was simulate the client with curl -d and the problem would have been pretty obvious.<p>And in this case the more complicated debugging tools didn&#x27;t even explain anything. As the last page says, the answer is kind of a leap. If you already knew about this problem curl would have solved it immediately, and if you didn&#x27;t you&#x27;d still be baffled even after knowing exactly why the stack traces are the way they are.

Really nice game&#x2F;tutorial.<p>The best job interview I ever had was framed like this. The interviewer told me there was a bug in the system and had a stack of pages he&#x27;d printed out that would provide successive clues as to what caused it. I could ask them questions, in effect using the interviewer as a search engine&#x2F;debugger.<p>It was the closest an interview has ever come to simulating the day-to-day of a web developer.

Author here. I wrote a post about the design of this game on my blog: <a href="https:&#x2F;&#x2F;jvns.ca&#x2F;blog&#x2F;2021&#x2F;04&#x2F;16&#x2F;notes-on-debugging-puzzles&#x2F;" rel="nofollow">https:&#x2F;&#x2F;jvns.ca&#x2F;blog&#x2F;2021&#x2F;04&#x2F;16&#x2F;notes-on-debugging-puzzles&#x2F;</a>

Knowledge of delayed ack and nagle&#x27;s algorithm wasn&#x27;t necessary to solve.<p>The explanation didn&#x27;t mention the flushHeaders call, which is apparently the fix. I didn&#x27;t run any tests, just looked at the JS and figured that sending 2 packets is worse than sending 1 w.r.t. latency.<p>It&#x27;s also a pretty strong intuition that the client side tends to have issues, since the server is usually well-tested and standardized. Also, very often people are measuring wrong, so checking the JS to be sure the time recorded is accurate is also important.<p><a href="https:&#x2F;&#x2F;nodejs.org&#x2F;api&#x2F;http.html#http_request_flushheaders" rel="nofollow">https:&#x2F;&#x2F;nodejs.org&#x2F;api&#x2F;http.html#http_request_flushheaders</a>

As requested by the commenter below (now I want spoiler tags):<p><i>SPOILER FOR THE GAME</i><p>I’ve seen TCP_NODELAY all over the place before, but never known why. This was a fun way to find out.

I got it straight away, but I have a telecoms engineering and networking background. This just goes to show how poorly networking is taught in most courses (as are databases), and specially in boot camps. Self-taught programmers are also very unlikely to be exposed to this kind of topic.<p>One thing that was not offered as an option was to use a packet analyzer like tcpdump or Wireshark, even though that is the most reliable and systematic way to get to the bottom of many performance problems. You&#x27;d think the popularity of the network tab in Chrome&#x27;s dev tools would make this less scary.

This is great. Julia Evans has some great content in general, she&#x27;s really good at explaining complex topics in an engaging and accessible way.

The interesting thing for me is that I would suspect most devs including myself would assume that if the request takes 50ms, that&#x27;s how long it takes (because networks!).<p>I wonder how many of us are able to judge how long something should take? Not me, except anecdotally.

I ran into an interesting variation of this where we shouldn&#x27;t have had any problems with small packets, but it turned out we had having jumbo frames enabled in AWS (which seems to be a default now). Together with gzip, you can actually have a bit of trouble filling up a packet, which will then be delayed by the commonly mentioned interaction with delayed ACKs.

Error: &lt;&lt;you-said&gt;&gt;: error within widget contents (Error: cannot find a closing tag for HTML &lt;pid&gt;)<p>This hints to a possible XSS and&#x2F;or code injection (not completely quoted input). Input was &quot;strace -s128 -f -p &lt;pid&gt;&quot; , as an answer to &quot;how do you strace server process&quot;

Also fun: try to tunnel TCP over TCP.

A comment from the inventor of Nagle’s algorithm: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=9050645" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=9050645</a><p>(tl;dr Try turning off delayed ACK first, especially if you can’t update the code.)

SPOILER ALERT<p>I answered &quot;req.flushHeaders()&quot; but surprisingly it doesn&#x27;t accept that as a cause, even though the headers would be sent with the initial packet and should improve the latency.

&quot;Also, the Linux kernel doesn&#x27;t always enable delayed ACKs -- to reproduce this I actually had to write a Python program that explicitly turns them on. I haven&#x27;t been able to find a clear explanation of exactly when delayed ACKs are used.&quot;<p>Delayed ACKs can be enabled on Linux kernels 3.11+ with ip(8).<p><pre><code> ip route change ROUTE quickack 1 </code></pre> On MacOS and Windows, delayed ACKs can be configured through sysctl and the registry, respectively.<p>Delayed ACKs may be used in response to congestion.<p>For example, in bulk, i.e., non-interactive, transfers with large packets, delayed ACKs can be useful.<p>This is covered in Chapters 15 (15.3) and 16 of Stevens&#x27; TCP&#x2F;IP Illustrated Vol. 1.<p>This draft suggests delayed ACKs are useful during TLS handshake.<p><a href="https:&#x2F;&#x2F;tools.ietf.org&#x2F;id&#x2F;draft-stenberg-httpbis-tcp-03.html" rel="nofollow">https:&#x2F;&#x2F;tools.ietf.org&#x2F;id&#x2F;draft-stenberg-httpbis-tcp-03.html</a><p>Also, socat allows for setting TCP options via setsockopt. No need to write a new program.

Twine and SugarCube! Interesting to see that pop up here:<p><a href="https:&#x2F;&#x2F;www.motoslave.net&#x2F;sugarcube&#x2F;2&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.motoslave.net&#x2F;sugarcube&#x2F;2&#x2F;</a>

That was exceptionally fun. I thought I had the answer but I was completely wrong. I shouldn&#x27;t have stopped the debugging and rush to the solution. Unfortunately it is a game, and it allowed me to do it.<p>To me this seems pretty obscure and you debug pretty deep into and outside of your application. One part of me thinks of this as Somebody Else&#x27;s Problem, but definitively makes me rethink it as a SEP and something devs should know about. Specially in time critical&#x2F;real time systems.

Based on the description, guessed &#x27;nagle&#x27; without any debugging. Not sure if it implies I was right or wrong, but it explained Nagle&#x27;s algorithm.<p>Asked what to do about it and typed &#x27;tcpnodelay&#x27;. It replied it wasn&#x27;t smart and asked me to click a button.<p>Feels pretty basic to anyone who&#x27;s ever really touched TCP in code.

A few months ago there had been multiple articles about this behaviour but i really don‘t remember the details anymore. Does anyone know a writeup with a detailled explanation to understand how it is happening and tests to see whether your systems are affected?

Loved it! Although it didn’t give a comprehensive answer on how to preventively solve the problem on any platform. Is there one? Or should software developers just stick to one way or another like some kind of an unspoken rule?

This was fun and a lesson to look at the code first before jumping into tcpdump&#x2F;tracing

A couple bugs in this game, I wanted to SSH onto the client, took me to the server instead.

Nice, but not perfect:<p>``` You said: &quot;strace -p $(ps aux| grep server.py| grep -v &quot;grep&quot;| awk -F &#x27; &#x27; &#x27;{print $2}&#x27;)&quot;.<p>To strace the server, first you need to find its PID. You know that the program is called server.py. ```

grr.. I felt stupid going through the puzzle.. but I was looking at that flushHeaders call and thought that might be a problem - simply because I never call that.

All due respect, this is a neat advertisment for the &quot;storytelling&quot; Javascript library she is using, but I learn much more by reading W R Stevens&#x27; books. There is more to TCP&#x2F;IP than what one can do through Berkeley sockets. Plus reading Stevens&#x27; books does not require Javascript.