Scala vs. Go TCP Benchmark

Did you consider running the go client against the scala server and vice versa?<p>Also, that&#x27;s kind of a lot of code. Here&#x27;s my rewrite of the server: <a href="http://play.golang.org/p/hKztKKQf7v" rel="nofollow">http:&#x2F;&#x2F;play.golang.org&#x2F;p&#x2F;hKztKKQf7v</a><p>It doesn&#x27;t return the exact same result, but since you&#x27;re not verifying the results, it is effectively the same (4 bytes in, 4 bytes back out). I did slightly better with a hand-crafted one.<p>A little cleanup on the client here: <a href="http://play.golang.org/p/vRNMzBFOs5" rel="nofollow">http:&#x2F;&#x2F;play.golang.org&#x2F;p&#x2F;vRNMzBFOs5</a><p>I&#x27;m guessing scala&#x27;s hiding some magic, though.<p>I made a small change to the way the client is working, buffering reads and writes independently (can be observed later) and I get similar numbers (dropped my local runs from ~12 to .038). This is that version: <a href="http://play.golang.org/p/8fR6-y6EBy" rel="nofollow">http:&#x2F;&#x2F;play.golang.org&#x2F;p&#x2F;8fR6-y6EBy</a><p>Now, I don&#x27;t know scala, but based on the constraints of the program, these actually all do the same thing. They time how long it takes to write 4 bytes * N and read 4 bytes * N. (my version adds error checking). The go version is reporting a bit more latency going in and out of the stack for individual syscalls.<p>I suspect the scala version isn&#x27;t even making those, as it likely doesn&#x27;t need to observe the answers.<p>You just get more options in a lower level language.

&gt; The experiments where performed on a 2.7Ghz quad core MacBook Pro with both client and server running locally, so as to better measure pure processing overhead. The client would make 100 concurrent connections and send a total of 1 million pings to the server, evenly distributed over the connections. We measured the average round trip time.<p>Another let&#x27;s rape localhost:8080 on a MacBook Pro™ benchmark

So even if Scala is ahead in this (flawed) benchmark, that&#x27;s not how you write a TCP server in Scala, because you want to do it non-blocking. Not doing it based on asynchronous I&#x2F;O means that in a real-world scenario the server will choke under the weight of slow connections, not to mention be susceptible to really cheap DoS attacks like Slowloris [1].<p>Seriously, it goes beyond the underlying I&#x2F;O API that you&#x27;re using. If anywhere in the code you&#x27;re reading from an InputStream or you&#x27;re writing to an OutputStream that&#x27;s connected to an open socket, then that&#x27;s a blocking call that can crush your server. Right now, every Java Servlets container that&#x27;s not compatible with the latest Servlets API 3.1 can be brought down with Slowloris, even if under the hood they are using NIO.<p>Option A for writing a server in Scala is Netty [2].<p>Option B for writing a server in Scala is the new I&#x2F;O layer in Akka [3].<p>[1] <a href="http://ha.ckers.org/slowloris/" rel="nofollow">http:&#x2F;&#x2F;ha.ckers.org&#x2F;slowloris&#x2F;</a><p>[2] <a href="http://netty.io/" rel="nofollow">http:&#x2F;&#x2F;netty.io&#x2F;</a><p>[3] <a href="http://doc.akka.io/docs/akka/snapshot/scala/io.html" rel="nofollow">http:&#x2F;&#x2F;doc.akka.io&#x2F;docs&#x2F;akka&#x2F;snapshot&#x2F;scala&#x2F;io.html</a>

&gt;The experiments where performed on a 2.7Ghz quad core MacBook Pro with both client and server running locally<p>No no no. Assuming your production code runs on Linux THAT is where you need to do this test. It is extremely naive to assume that either the JVM or the Go runtime will perform system interfacing tasks even remotely similar between OSX and Linux. Linux is what you will use in production. Linux is almost always faster (more effort from devs both on the kernel TCP&#x2F;IP side and the runtime&#x2F;userspace side).<p>Write your Scala, Java, Go wherever you want, but please, benchmark it in a clone of your production environment!<p>P.S. In production I assume your client and server will not be local... don&#x27;t do this, kernels do awesome&#x2F;dirty optimizations over loop-back interfaces, sometimes even bypassing large parts of the TCP&#x2F;IP stack, parts you want included in any meaningful benchmark.

Results from my Mac:<p><pre><code> Go server vs Go client: 10ms Scala server vs Scala client: 3ms Go server vs Scala client: 4ms Scala server vs Go client: ???? </code></pre> Scala server against the Go client is really slow (?). I reduced the ping count by a factor of 100, and extrapolating I think it would have reported around 670ms. What gives?<p>I don&#x27;t know much about Scala Futures, but isn&#x27;t Scala&#x27;s client doing something completely different than the Go client? Scala&#x27;s client with Future.sequence looks like it&#x27;s calling each `ping` method sequentially.<p>Printing the connection identifier {0,100} on open &amp; close shows that while it isn&#x27;t completely sequential, only about a handful of connections are open at a time.<p>On the other hand, the Go client appears to switch amongst goroutines more frequently. All the connections open before any connection closes.<p>In other words, the I think the difference in performance is due to the difference in how randomly the connections are shuffled. The terrible performance time in the last case I think shows a bottleneck in the Scala server rather than the Go client.

Isn&#x27;t this more like a comparison of the JVM&#x27;s TCP library vs Go&#x27;s TCP library... not so much Scala vs Go?

If the eventual production app would run on Linux (which I&#x27;m only guessing based on the context), this benchmark should probably be run there. Darwin&#x27;s surprisingly higher system call and context switch overhead can be deceptive for apps that are OS-bound.

Quick look at the language benchmark games, and Go is not 10x slower than Java for most tests. Java is often 3-5x slower than C&#x2F;C++&#x2F;etc (although maybe it didn&#x27;t get enough time to warm up).

Ok, so. When you need to write a load balancer and want to test different languages for the task, you don&#x27;t do a benchmark like that one.<p>Writing a &quot;ping-pong&quot;? And not using the same client to test both servers?<p>It would not have been too hard to write a simple proxy in both languages. Not even worrying about parsing HTTP headers, just testing TCP, that is really easy.<p>Now, if you really want to test the performance, you have to implement it differently. Just two small features you would need in both:<p>* non blocking IO: right now, you&#x27;re starting a new future in Scala, that&#x27;s easy to write but not really efficient (it might work better with goroutines) * zero copy: if you&#x27;re load balancing, you will spend your time moving bits, so you&#x27;d better make sure that you don&#x27;t copy them too much. It is possible with Scala, but it looks like Go does not support it<p>Now, when you have reasonable testing grounds (that woudn&#x27;t be more that a hundred lines in both languages), better get your statistics right.<p>&quot;The client would make 100 concurrent connections and send a total of 1 million pings to the server, evenly distributed over the connections. We measured the average round trip time&quot; -&gt; that is NOT how you should test. Here, you would want to know what is the nominal pressure the balancer could handle, so you must measure a lot of metrics:<p>* RTT * bandwidth (per user and total) * time to warmup (the JVM optimizes a lot of things on the fly, you have to wait for it) * operational limits (what is the maximal bandwidth for which the performance crashes? same for number of users)<p>And then, you don&#x27;t measure only the average values. You must measure the standard deviation. Because you could have a good average, but wildly varying data, and that is not good.<p>Last thing: the macbook may not be a good testing system.

What would be the advantages of using a custom built load balancer vs. something off the shelf like Nginx?

The benefit of go is the cheap threads and CSP which make it scale well for complex servers.<p>I think we&#x27;ll see the Go runtime gaining Single System Image distribution, performance improvements and libraries not services (e.g. groupcache) making it a very different world to develop in than Scala.

<i>... that the Go server had a memory footprint of only about 10 MB vs. Scala’s nearly 200 MB.</i> Priceless!<p>200Mb for such a crappy server with only hundred connections.<p>So, it is not about whose wrapper around epoll is thinner, but about how the data are represented and copied.

I think the JVM TCP stack is extremely mature, but I&#x27;m really surprised to see that Go is ten times slower.<p>It would have been interesting to have a C&#x2F;C++ benchmark for reference.

&quot;The actual test code contained some functionality to deal with connection errors, omitted here for brevity.&quot;<p>Any way to see the actual test code?

Ugh, after reading all the comments here I wonder how a mere-mortal programmer gets multi-threaded, network programming done right. It&#x27;s not clear to me if there is a clear winner between Go and Scala&#x2F;JVM. Are the majority of programs out there crappy, memory-hogging and non-performant?<p>EDIT: Any good references out there? Thanks!

So this Go server is slower than a single threaded, blocking TCP server in Ruby. And the memory? Almost the same:<p>require &quot;socket&quot; s = TCPServer.new &#x27;localhost&#x27;, 1201 loop do c = s.accept if c.read(4) c &lt;&lt; &#x27;Pong&#x27; end c.close end

It is the JVM rather than Scala - one of the fastest VMs implemented till date.

As a word of advice -- you are almost certainly wasting your time writing a load balancer. There are close to zero cases where someone can legitimately justify such an exercise.<p>In any case, your benchmark is flawed (as virtually all benchmarks are). The reason Go <i>the client</i> is slower is because it tries to pervasively &quot;thread&quot; via the M:N scheduler -- every wait check causes it to yield the actual thread and switch goroutine, creating a large amount of overhead. The Scala cases, on the other hand, is dramatically more limited and will not yield this overhead.<p>The Go server does not have this fault, and is likely top performance. And aren&#x27;t we talking about a server anyways?<p>Now as to the client, while we could naively criticize M:N scheduling based upon this, try giving it a more realistic workload (unless you seriously plan on load balancing pongs): Instead of ping&#x2F;pong, return larger lengths of data preferably over actual network connections (not localhost) - e.g. 32KB.<p>The Go client will catch up if not shoot into the lead. M:N scheduling is optimal for most real-world workloads, though it is less optimal for spin-off-a-million-goroutines that do nothing type tests.<p>This is not a test of TCP overhead, or a realistic test, but instead demonstrates the small overhead of goroutines when you give each a minuscule amount of wait work.

EDIT. OK, so. I ran this benchmark myself on an 8-core Xeon running Linux. 2.13 GHz, CentOS 6.2. Kernel was 2.6.32-220.el6.x86_64. 50 gigs of RAM.<p>I got somewhere between 2.0 and 2.2 &quot;milliseconds per ping&quot; for Scala 2.9, and somewhere between 3.5 and 3.7 for Go 1.1. This is not the 10x difference that the authors reported, but it is something. The difference may be due in part to the different platform and hardware I am using.<p>Contrary to what I wrote earlier, I noticed that GOMAXPROCS=8 did seem to be slower than GOMAXPROCS=4 here. I got around 4 &quot;milliseconds per ping&quot; with GOMAXPROCS=8. Using a mutex and explicit condition variable shaved off maybe 0.2 milliseconds on average (very rough estimation).<p>Again contrary to what I wrote earlier, Nagle on versus off didn&#x27;t seem to matter in the Go code. I still think you should always have it off for a test like this, but on my setup I did not see a difference.<p>I still don&#x27;t think this benchmark is showing what they think it is. I have a hunch that this is more of a scheduler benchmark than a TCP benchmark at all. I think I&#x27;d have to haul out vtune to get any further, and I&#x27;m getting kind of tired (after midnight here).