Node.js in Flame Graphs

The moneyquote:<p><i>&quot;We made incorrect assumptions about the Express.js API without digging further into its code base. As a result, our misuse of the Express.js API was the ultimate root cause of our performance issue.&quot;</i><p>This situation is my biggest challenge with software these days. The advice to &quot;just use FooMumbleAPI!&quot; is rampant and yet the quality of the implemented APIs and the amount of review they have had varies all over the map. Consequently any decision to use such an API seems to require one first read and review the entire implementation of the API, otherwise you get the experience that NetFlix had. That is made worse by good APIs where you spend all that time reviewing them only to note they are well written, but each version which could have not so clued in people committing changes might need another review. So you can&#x27;t just leave it there. And when you find the &#x27;bad&#x27; ones, you can send a note to the project (which can respond anywhere from &quot;great, thanks for the review!&quot; to &quot;if you don&#x27;t like it why not send us a pull request with what you think is a better version.&quot;)<p>What this means in practice is that companies that use open source extensively in their operation, become slower and slower to innovate as they are carrying the weight of a thousand different systems of checks on code quality and robustness, which people using closed source will start delivering faster and faster as they effectively partition the review&#x2F;quality question to the person selling them the software and they focus on their product innovation.<p>There was an interesting, if unwitting, simulation of this going on inside Google when I left, where people could check-in changes to the code base that would have huge impacts across the company causing other projects to slow to a halt (in terms of their own goals) while they ported to the new way of doing things. In this future world changes, like the recently hotly debated systemd change, will incur costs while the users of the systems stop to re-implement in the new context, and there isn&#x27;t anything to prevent them from paying this cost again and again. A particularly Machievellan proprietary source vendor might fund programmers to create disruptive changes to expressly inflict such costs on their non-customers.<p>I know, too tin hat, but it is what I see coming.

&gt; It’s unclear why Express.js chose not to use a constant time data structure like a map to store its handlers.<p>Its actually quite clear - most routes are defined by a regex rather than a string, so there is no built-in structure (if there&#x27;s a way at all) to do O(1) lookups in the routing table. A router that only allowed string route definitions would be faster but far less useful.<p>I can&#x27;t explain away the recursion, though. That seems wholly unnecessary.<p>Edit: Actually, I figured that out, too. You can put middleware in a router so it only runs on certain URL patterns. The only difference between a normal route handler and a middleware function is that a middleware function uses the third argument (an optional callback) and calls it when done to allow the route matcher to continue through the routes array. This can be asynchronous (thus the callback), so the router has to recurse through the routes array instead of looping.

I&#x27;m surprised nobody has mentioned that express has a built in mechanism for sublinear matching against the entire list of application routes. All you have to do is nest Routers (<a href="http://expressjs.com/4x/api.html#router" rel="nofollow">http:&#x2F;&#x2F;expressjs.com&#x2F;4x&#x2F;api.html#router</a>) based on URL path steps and you will reduce the overall complexity of matching a particular route from O(n) to near O(log n).

I wonder what the thought process was behind moving their web service stack (partially?) to node.js in the first place. For a company with the scale and resources of Netflix it&#x27;s not exactly an obvious choice.

TIL, SVG&#x27;s can display labels on element hover: <a href="http://cdn.nflximg.com/ffe/siteui/blog/yunong/200mins.svg" rel="nofollow">http:&#x2F;&#x2F;cdn.nflximg.com&#x2F;ffe&#x2F;siteui&#x2F;blog&#x2F;yunong&#x2F;200mins.svg</a><p>Nice, contained way to show data like this.

&gt; ...as well as increasing the Node.js heap size to 32Gb.<p>&gt; ...also saw that the process’s heap size stayed fairly constant at around 1.2 Gb.<p>This is because 1.2 GB is the max allowed heap size in v8. Increasing beyond this value has no effect.<p>&gt; ...It’s unclear why Express.js chose not to use a constant time data structure like a map to store its handlers.<p>It it is non-trivial (not possible?) to do this in O(1) for routes that use matching &#x2F; wildcards, etc. This optimization would only be possible for simple routes.

Sounds like a documentation issue, or lack of a staging environment. I&#x27;ve written and maintained countless large Express applications and routing was never even remotely a bottleneck, thus the simple &amp; flexible linear lookup. I believe we had an issue or two open for quite a while in case anyone wanted to report real use-cases that performed poorly.<p>Possibly worth mentioning, but there&#x27;s really nothing stopping people from adding dtrace support to Express, it could easily be done with middleware. Switching frameworks seems a little heavy-handed for something that could have been a 20 minute npm module.

I read:<p>&quot;This turned out be caused by a periodic (10&#x2F;hour) function in our code. The main purpose of this was to refresh our route handlers from an external source. This was implemented by deleting old handlers and adding new ones to the array&quot;<p><i>refresh our route handlers from an external source</i><p>This is not something that should be done in live process. If you are updating the state of the node, you should be creating a new node and killing the old one.<p>Aside from hitting a somewhat obvious behavior for messing with the state of express in running process, once you have introduced the idea of programmatically putting state into your running node you have seriously impeded the abiltity to create a stateless fault tolerant distributed system.

&gt; benchmarking revealed merely iterating through each of these handler instances cost about 1 ms of CPU time<p>1ms &#x2F; entry? What is it doing that it&#x27;s spending 3 million cycles on a single path check?

&gt; I can’t imagine how we would have solved this problem without being able to sample Node.js stacks and visualize them with flame graphs.<p>This has me scratching my head. The diagrams are pretty, maybe, but I can&#x27;t read the process calls from them (the words are truncated because the graphs are too narrow). And I can&#x27;t see, visually, which calls are repeated. They&#x27;re stacked, not grouped, and the color palette is quite narrow (color brewer might help here?).<p>At least, I _can_ imagine how you could characterize this problem without novel eye-candy. Use histograms. Count repeated calls to each method and sort descending. Sampling is only necessary if you&#x27;ve got -- really, truly, got -- big data (which Netflix probably does), but I don&#x27;t think the author means &#x27;sample&#x27; in a statistical sense. It sounds more like &#x27;instrumentation&#x27;, decorating the function calls to produce additional debugging information. Either way, once you have that, there are various common ways to isolate performance bottlenecks. Few of which probably require visual graphs.<p>There&#x27;s also various lesser inefficiencies in the flame graphs: is it useful (non-obvious) that every call is a child of `node`, `node::Start`, `uv_run`, etc.? Vertical real-estate might be put to better use with a log-scale? Etcetera, etc.

<p><pre><code> &gt; our misuse of the Express.js API was the &gt; ultimate root cause of our performance issue </code></pre> That&#x27;s unfortunate. Restify is a nice framework too, but mistakes can be made with any of them. Strongloop has a post comparing Express, Restify, hapi and LoopBack for building REST API&#x27;s for anyone interested. <a href="http://strongloop.com/strongblog/compare-express-restify-hapi-loopback/" rel="nofollow">http:&#x2F;&#x2F;strongloop.com&#x2F;strongblog&#x2F;compare-express-restify-hap...</a>

From the article:<p>&gt; What did we learn from this harrowing experience? First, we need to fully understand our dependencies before putting them into production.<p>Is that the lesson to learn? That scares me, because a) it&#x27;s impossible, and b) it lengthens the feedback loop, decreasing systemic ability to learn.<p>The lesson I&#x27;d learn from that would be something like &quot;Roll new code out gradually and heavily monitor changes in the performance envelope.&quot;<p>Basically, I think the approach of trying to reduce mean time between failure is self-limiting, because failure is how you learn. I think the right way forward for software is to focus on reducing incident impact and mean time to recovery.

My biggest takeaway from this article is that Netflix is moving from Express to Restify, and I look forward to watching the massive uptick this has on <a href="https://github.com/mcavage/node-restify/graphs/contributors" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;mcavage&#x2F;node-restify&#x2F;graphs&#x2F;contributors</a>

If I had to pick one line to highlight (not to criticize, but was a wise lesson worth sharing) it would be this one:<p>&quot;First, we need to fully understand our dependencies before putting them into production.&quot;

I would like to mention that Netflix could have consulted the express maintainers (us) but didn&#x27;t.<p>Source: myself - <a href="https://github.com/strongloop/express/pull/2237#issuecomment-59681175" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;strongloop&#x2F;express&#x2F;pull&#x2F;2237#issuecomment...</a>

A surprising amount of path recognizers are O(n). Paths&#x2F;routes are a great fit for radix trees, since there&#x27;s typically repetitions, like &#x2F;projects, &#x2F;projects&#x2F;1, and &#x2F;projects&#x2F;1&#x2F;todos. The performance is O(log n).<p>I built one for Java: <a href="https://github.com/augustl/path-travel-agent" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;augustl&#x2F;path-travel-agent</a>

tl;dr:<p>* Netflix had a bug in their code.<p>* But Express.js should throw an error when multiple route handlers are given identical paths.<p>* Also, Express.js should use a different data structure to store route handlers. EDIT: HN commentors disagree.<p>* node.js CPU Flame Graphs (<a href="http://www.brendangregg.com/blog/2014-09-17/node-flame-graphs-on-linux.html" rel="nofollow">http:&#x2F;&#x2F;www.brendangregg.com&#x2F;blog&#x2F;2014-09-17&#x2F;node-flame-graph...</a>) are awesome!

It&#x27;s not just the extra lookups -- static in express is deceptively dog-slow. For every request it processes, it stats every filename that might satisfy the URL. This results in an enormous amount of useless syscall&#x2F;IO overhead. This bit me pretty hard on a high-throughput webservice endpoint with an unnoticed extra static middleware. I wound up catching it with the excellent NodeTime service.<p>Now that I look at it, there&#x27;s a TOCTOU bug on the fstat&#x2F;open callback, too: <a href="https://github.com/tj/send/blob/master/index.js#L570-L605" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;tj&#x2F;send&#x2F;blob&#x2F;master&#x2F;index.js#L570-L605</a><p>This should be doing open-then-fstat, not stat-then-open.

I am upset that the title has been changed from &quot;Node.js in Flames&quot;. Which is not only the real title of the article, but also a reasonable description of what they&#x27;ve been facing with Node.<p>#moderationfail

This is the first I&#x27;ve heard of restify, but it seems like a useful framework for the main focus of most Node developers I know, which is to replace an API rather than a web application.

&gt; This turned out be caused by a periodic (10&#x2F;hour) function in our code. The main purpose of this was to refresh our route handlers from an external source. This was implemented by deleting old handlers and adding new ones to the array. Unfortunately, it was also inadvertently adding a static route handler with the same path each time it ran.<p>I don&#x27;t understand the need of refreshing route handlers. Could someone explain they needed to do this, and also why from an external source?

The express router array is pretty easy to abuse, it&#x27;s true. For example, as something you probably shouldn&#x27;t ever do:<p><a href="https://www.exratione.com/2013/03/nodejs-abusing-express-3-to-enable-late-addition-of-middleware/" rel="nofollow">https:&#x2F;&#x2F;www.exratione.com&#x2F;2013&#x2F;03&#x2F;nodejs-abusing-express-3-t...</a><p>I guess the Netflix situation is one of those that doesn&#x27;t occur in most common usage; certainly dynamically updating the routes in live processes versus just redeploying the process containers hadn&#x27;t occurred to me as a way to go.

Responses are already firing in: <a href="https://news.ycombinator.com/item?id=8632220" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=8632220</a>

I love these kinds of investigations into problems in production. I mean, you really have to admire their determination in getting to the root of the problem.<p>In some ways, these engineers are not that different from academic researchers, in that they are devising experiments, verifying techniques, all in the pursuit of the question: why?

I would have written my apis in golang and not nodejs. Go is way faster in my experience and it feels leaner to create something because creating a web service can be productively doneout of box. Node apps tend to depend on thousands of 3rd party dependencies which makes the whole thing feel fragile to me.

Would someone explain what I&#x27;m missing about the flame graphs? Why are they indispensable here? In a normal profiler, you&#x27;d just expand the hot path and see what had the most samples. Apart from making recursion very explicit, what special aspect do flame graphs expose?

Why are they loading in routes from an external source? Is that normal, I have never seen that before.

Interesting article. I have a lot of experience dealing with ETLs in WPA on the Windows side - it&#x27;s an awesome tool that gives you similar insights. I haven&#x27;t used it for looking at javascript stacks before though, so I don&#x27;t know if it&#x27;ll do that.

&gt; We also saw that the process’s heap size stayed fairly constant at around 1.2 Gb.<p>&gt; Something was adding the same Express.js provided static route handler 10 times an hour.<p>Why didn&#x27;t it increase the heap size? Maybe it was too small to be noticeable?

<i>Second, given a performance problem, observability is of the utmost importance</i><p>I couldn&#x27;t agree with this more. Understanding where time is being spent and where pools etc. are being consumed is critical in these sorts of exercises.

So the lesson is to actually know the code you deploy to prod? Is that not obvious?

NodeJS Project has already a similar issue about recursive route matching.<p><a href="https://github.com/strongloop/express/issues/2412" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;strongloop&#x2F;express&#x2F;issues&#x2F;2412</a>

Doesn&#x27;t this seem like a bug in the express router? All of the additional routes in the array are dead (can&#x27;t be routed to).

I wonder how Netflix would perform with using Dart with the DartVM. I reckon it would be faster than Node based on benchmarks I&#x27;ve seen. Chrome DartVM support is right around the corner ;)

Crazy talk. In 1ms, I can perspective transform a moderately big image. NodeJS cant iterate through a list.<p>We really need a 60 fps equivalent for web stuff. You have 16ms, thats it.

I must admit I could enjoy just doing this type of analysis all day long. Yet I hate non computing puzzles.

wow. I love that Netflix us using Node and even more curious that they would use express.

this is why you stick to tried and true methods folks. this is such a typical node.js fanboy mentality. &quot;reinventing the wheels is justified because asynchronous&quot;. or &quot;i want this trendy way to do things just because everyone else is jumping on the bandwagon&quot;.<p>Give me flask + uwsgi + nginx anyday.

an unfortunate title. Ha ha &quot;flames&quot; ha ha &quot;Node.js&quot; but the article is really about express. Not so &quot;ha ha&quot;

A very good reason to go with express is TJ. He was the initial author of express and he is quite brilliant when it comes to code quality. Of course, TJ is no more part of the community but his legacy lives :-)

OFFTOPIC: &quot;Today, I want to share some recent learnings from performance tuning this new application stack.&quot;<p>The word you want is &quot;lessons&quot;.