I might get flamed/downvoted for this, but where's the content of this article? Apart from some vague tropes ("Erlang is bad at string processing!" "We need scalability, and that means async Python", "Pick the right Python libraries!"), there's really nothing interesting about the implementation details, especially considering such a attention-grabbing headline (to put it nicely).<p>What <i>specifically</i> was bad about the Erlang code? Isn't this just saying that if nobody in your company really understands a language, don't use it?<p>This is more about the technical competency of a specific company than general technical issues. Or, to put it bluntly, it's more "Mixpanel sucks at Erlang" than "Erlang sucks". Don't get me wrong, I'd be really, really interested in a good analysis why in this case Erlang was the wrong choice, but this article didn't even get close to anything <i>technically</i> interesting (Even with the ubiquitous requests/s graph).
Pretty harsh response to a post from an intern, dudes and dudettes.<p>Yeah, the erlang server code was pretty gross. It was one of the very first things written when we started Mixpanel over 2 years ago, and it's only been updated a few times since then.<p>I feel like the big thing you guys are missing is how little <i>time</i> we have. It's not like we don't know when we have bad code, or we don't realize that we made mistakes in the initial server design - we just have a million things on our collective plate. Fixing a very simple server - (accept request, validate json, put on queue) that is doing its job okay hasn't been a high priority thing.<p>When this code was written, Mixpanel had zero customers and we weren't sure what we were building yet. In that regard, Erlang has been a rock. We've barely had to touch it during the rampup from 0 to thousands of requests per second.<p>Now that we have the manpower, and we've learned what we really need, we can rewrite it to make things easier on ourselves. If we can get acceptable performance in python, there is no reason to use erlang.<p>I think there's some merit to the other complaints (error codes, etc), but that's another symptom of this thing being written so long ago. We want to improve things incrementally (and backwards-compatibly) for now, but it will be dramatically simpler for us to make changes to the server now that it's written in python.<p>Ultimately, we have skeletons in the closet, just like the rest of you - I'm sure <i>all</i> of you have some bad code in production somewhere. Now we're saying "Look, we're getting rid of our skeletons!" and you guys are like "OMG WHY YOU HAVE SKELETONS" instead of "sweet, no more skeletons".
<p><pre><code> Finally, we use a few stateful, global data structures to
track incoming requests and funnel them off to the right
backend queues. In Erlang, the right way to do this is to
spawn off a separate set of actors to manage each data
structure and message pass with them to save and retrieve
data. Our code was not set up this way at all, and it was
clearly crippled by being haphazardly implemented in a
functional style.
</code></pre>
Seriously?! I have used only a little erlang, but this makes no sense to me - it's like you were writing some big java project and put everything in one huge class, with all methods and variables static. It's hard for me to imagine why and how someone would write production erlang app with no actors, especially some kind of server. No wonder the thing sucked in the first place.
From the article:<p><pre><code> Because of these performance requirements, we originally wrote the
server in Erlang (with MochiWeb) two years ago. After two years of
iteration, the code has become difficult to maintain. No one on
our team is an Erlang expert, and we have had trouble debugging
downtime and performance problems. So, we decided to rewrite it
in Python, the de-facto language at Mixpanel.
</code></pre>
My first impulse would have been to have one or more team members become Erlang experts. Was that considered?
“Finally, we use a few stateful, global data structures to track incoming requests and funnel them off to the right backend queues. In Erlang, the right way to do this is to spawn off a separate set of actors to manage each data structure and message pass with them to save and retrieve data.”<p>Nope, that’s not the right way. The way you were doing it ended up making all calls sequential and bound to single processes that could lose state. That’s not right.<p>The best way to do it would have been to use ETS tables (which can be optimized either for parallel reads or writes), which also allows destructive updates, in order to have the best performance and memory usage possible. Note that you could then have had memory-only Mnesia table (adding transactions, sharding and distribution on top of ETS) to do it.<p>As for string performances, I’m wondering if you used lists-as-strings, binary strings or io-lists to do your things. This can have significant impact in performance and memory use.<p>Then again, if you had a bunch of Python and no Erlang experts, I can’t really say anything truly convincing against a language switch. Go for what your team feels good with.
> The biggest challenge for me was pushing the server from working 99.9% of the time to 99.99% of the time, because those last few bugs were especially hard to find.<p>Could you expand upon this some more? How do you know the server works 99.99% of the time (or 99.9%)? Do you run regression tests using actual past requests?
It sounds like they're accepting really simple HTTP requests (event updates) and inserting a job in a queue.<p>Really simple + rarely changing + needs to scale to really high req/sec = perfect candidate for being written in C. Maybe as an nginx module?
"The main difference is that eventlet can’t influence the Python runtime, but actors are built into Erlang at a language level, so the Erlang VM can do some cool stuff like mapping actors to kernel threads (one per core) and preemption. We get around this problem by launching one API server per core and load balancing with nginx."<p>The actor model is for <i>concurrency</i>, which is when your threads are communicating with one another, right? What about the task that the API server does requires inter-thread communication?
The author is wrong about simplejson performing 10x better than the json included with python.<p>Here is my proof: <a href="http://j2labs.tumblr.com/post/7305664569/python-vs-javascript-the-json-race" rel="nofollow">http://j2labs.tumblr.com/post/7305664569/python-vs-javascrip...</a>
There's not much about "why," in fact, these are the only sentences that are at all relevant to "why:"<p><i>After two years of iteration, the code has become difficult to maintain. No one on our team is an Erlang expert, and we have had trouble debugging downtime and performance problems.</i><p><i>Erlang is historically bad at string processing, and it turns out that string processing is very frequently the limiting factor in networked systems because you have to serialize data every time you want to transfer it. There’s not a lot of documentation online about mochijson’s performance, but switching to Python I knew that simplejson is written in C, and performs roughly 10x better than the default json library.</i><p><i>I was able to provide some important operations in constant time along with other optimizations that were cripplingly slow in the Erlang version.</i><p><i>The [Python] community is extremely active, so many of my questions were already answered on Stack Overflow and in eventlet’s documentation.</i>
It seems like the lesson here is that basically any language (Python, Ruby, ...) will perform about the same with non-blocking I/O.<p>Does this mean that Erlang and node.js are mostly compelling because of the prevalence of async versions of common libraries? Or are they not that compelling in web contexts in the first place?
After 2 years in production you nave no one on the inside that knows the core part of your system? Duh! Start investing time in your core technology. Blaming Erlang for poor R&D management choices is not going to fly here.
Do they run the message queue on the same box as the gateway server in production? If not then the test he ran isn't a direct comparison (since network latency between the app server & queue isn't accounted for). Running both of those services on the same box isn't great either, since they could slow each other down, and you lose both if the box dies.<p>Still, very cool, congrats ankrgyl, it's awesome to be able to write stuff like that as an intern!
This sums it up:<p><pre><code> "No one on our team is an Erlang expert"
</code></pre>
Regarding mochijson we switched to jiffy [1] (NIF-based native C parser).<p>Also I would love to get a comparison between 2-years old (probably badly written) Erlang server and a new Python/eventlet server.<p>[1] <a href="https://github.com/davisp/jiffy" rel="nofollow">https://github.com/davisp/jiffy</a>
related: a benchmark of mochiweb vs cowboy vs misultin (all Erlang) vs node.js vs Python Tornado:<p><a href="http://www.ostinelli.net/a-comparison-between-misultin-mochiweb-cowboy-nodejs-and-tornadoweb/" rel="nofollow">http://www.ostinelli.net/a-comparison-between-misultin-mochi...</a>
Riak is a fairly large, open source, NoSQL database, written in erlang. I've looked at its source code on occasion knowing little about its internals, and found them to be really comprehensible. Sometimes it is shocking to see how elegant the code is.<p>At the same time, I have gone and looked at code I wrote back when I was first looking at erlang, that does much less and is much more verbose, confusing and sprawling.<p>I don't think erlang lacks maintainability. I think it just requires some discipline- like any language.<p>It sounds like your company has a culture of python hackers and erlang was chosen because you felt you needed to choose something "serious" for this bit of work, rather than because you loved erlang and would use erlang even if you needed to write something trivial. There's nothing wrong with that, but I don't see this article as revealing any hidden weaknesses in erlang.<p>Regarding the JSON parsing issue, erlang has excellent support for code written in other languages, specifically C, and you could wrap any C based JSON parser and use it, though I bet someone has already done this for you. I believed I was watching such a project on GitHub but can't for the life of me find it now.