Plurk Comet: Handling of 100.000+ open connections

Being a python programmer I am a bit worried that one (In this case) needs to use java to overcome a scaling problem. From my experiences with twisted, which is on much smaller scale than the article, I have never found twisted to be anykind of bottleneck.<p>It may be that since twisted(python) can only use one cpu effectively per interpreter(GIT et al) that it got left behind java which can easily use multiple CPUs for threads.<p>A slightly different architecture might be required then where multiple python processes are used.

This is a nice writeup that gives some good insight into what you're up against if you want to implement Comet today. It's a cool idea, but there's really nothing out there that you can use to do it off the shelf. It's great that he's gone through and actually tested out a bunch of these candidate technologies under load, so the next guy will at least know what not to try.<p>I have faith that Comet will have its day eventually, but clearly it's not quite time. With the current crop of production quality web servers, handling 100k requests per second (polling) is a solved problem. Keeping 100k simultaneous connections alive (comet) is by no means solved. It'll be cool when it happens though.

This article seems relevant, BTW:<p><a href="http://news.ycombinator.com/item?id=334506" rel="nofollow">http://news.ycombinator.com/item?id=334506</a>

SPEED :: In defence of Java, it is very quick when done right. See Kilim[1] for examples.<p>SUPPORT :: On top of the speed Java has a supported hardware stack (Solaris). So if doing 100K connections is <i>important</i> to you, you can find engineers who can diagnose difficult problems on your whole stack.<p>--------------<p>[1] <a href="http://www.youtube.com/watch?v=37NaHRE0Sqw" rel="nofollow">http://www.youtube.com/watch?v=37NaHRE0Sqw</a> at 23:23 . Java is quick without Kilim too, this is just an example with stats.

Has anyone had any success with orbited?

There was a nice write up on Erlang based Comet apps here: <a href="http://cometdaily.com/2007/12/14/getting-started-with-comet-on-erlang/" rel="nofollow">http://cometdaily.com/2007/12/14/getting-started-with-comet-...</a><p>Has anybody tried comet using Yaws?

Number of open connections is a little bit meaningless until they're actually under load, sending and receiving packets in a real world situation.<p>I'd agree, Java+NIO is a great system. Extremely fast+scalable.

<a href="http://www.alexa.com/siteinfo/douban.com" rel="nofollow">http://www.alexa.com/siteinfo/douban.com</a><p>AFAIK douban.com seems to be the largest (pure) Python site.

Hmm, was twisted not the next big thing that everyone was rewriting their app in? What does he mean, it does not scale?

For the non UK people 100.000 = 100,000 or 100 000 if you want the ISO version.

Interesting. I'm writing a comet server right now, using Ruby's EventMachine. I might do some tests to see how many connections I can stretch it out to; unfortunately I have the feeling that local connections are not going to quite match up to real live outgoing ones.<p>If I was doing it For Realz™ I reckon Erlang would be the go, still haven't gotten up to anywhere near that level of proficiency though. 37Signals recently rewrote their push server in Erlang and reported great results.

Is an actual c++ implementation of the java's nio is strictly better than nginx's one? Is it using epool() and sendfine()? Isn't all those java's abstraction leyers are overhead?<p>It seems like yet another attempt to put some more air into java's bubble.