TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

Frameworks Round 3

92 pointsby amarsahinovicabout 12 years ago

25 comments

jcobyabout 12 years ago
A few things really stand out for me:<p>1) I am shocked how slow EC2 is and how expensive. A m1.large is $0.240/hr or ~$175/month. And it's 7-10x slower than a $350/mo dedicated box. You would be spending $1500/mo to equal one dedicated box (not including bandwidth and S3 fees). A reserved instance is cheaper of course.<p>2) The multiple queries test would seem to be the best one to really simulate real-world usage. JSON-serialization is mostly testing the language.<p>This test pretty much puts most of the interpreted languages on a full stack framework together at the bottom. Although django seems to come out at the bottom of the pack for some reason.<p>Then come the "raw" tests and JIT languages running full stack frameworks. And at the top are the compiled languages. Not really surprising there.<p>3) I'm surprised php-raw did so well. And that go did so poorly.
评论 #5590389 未加载
评论 #5590274 未加载
评论 #5590361 未加载
评论 #5591063 未加载
zixabout 12 years ago
Honest benchmarks. Open and gracious group of developers behind it. Disclaimer - I'm married to one of these guys. I'm not a programmer, but I can see how hard this group has worked to provide something useful to the community and I've enjoyed watching it grow.
评论 #5590753 未加载
评论 #5590161 未加载
error54about 12 years ago
Just curious but why are the major php frameworks so abysmally slow? It seems like php-raw does pretty well in all the tests but most of the big frameworks (cake, symfony, etc) are orders of magnitude slower.
评论 #5590148 未加载
评论 #5590192 未加载
评论 #5590261 未加载
评论 #5590362 未加载
pilgrim689about 12 years ago
I'm having a hard time making conclusions based off of this data, except that Python/Ruby/JavaScript are slow, JVM is fast, and the rest are in between... It's odd how Haskell (Yesod) and C (Onion) are both slower than the JVM frameworks. Is it just a maturity issue? I know Yesod is very new relative to some of these JVM frameworks.<p>Also, minor typo: the link to Erlang from the Environment tab is not the right URL. You probably meant to reference <a href="http://www.erlang.org/" rel="nofollow">http://www.erlang.org/</a>, not <a href="http://www.erlang.com/" rel="nofollow">http://www.erlang.com/</a>
评论 #5590760 未加载
评论 #5591090 未加载
评论 #5590648 未加载
bhauerabout 12 years ago
Thanks amarsahinovic for sharing this here. This is a brief blog entry with some observations about Round 3 of our web frameworks benchmark project. The community contributions continue unabated and I think the impressively long charts demonstrate just how much the community has given us. Thank you to everyone who has participated and we are looking forward to Round 4 already!<p>Please reply to let us know if you have any comments, questions, or criticisms. We'd love to hear your feedback.
评论 #5590151 未加载
评论 #5590055 未加载
bdesimoneabout 12 years ago
Something seems off about "Results from dedicated hardware." Play and Go went from 1000s to &#60;10 respectively. Care to give any insight there?
评论 #5590132 未加载
dbeckerabout 12 years ago
Not being a web programmer, I have a naive question:<p>Are many websites running into speed or scalability issues?<p>I see programmers in other areas spend a lot of man-hours optimizing code that isn't a meaningful bottleneck. Is the same thing happening here?
评论 #5590359 未加载
xb95about 12 years ago
I am not acquainted with what language every framework uses. It would be useful for me in interpreting the data if you could annotate -- Java/JVM, PHP, Python, Perl, etc.<p>This is great work, though. Thank you for doing it -- and double thank you for taking submissions from each of the communities so they can try to submit their best shot. That really helps make sure the tests are as ideal as possible.
评论 #5590817 未加载
josh-jabout 12 years ago
Is the following a fair generalization?<p>Java should be the conservative choice for your web framework's language (rather than PHP). Scala, Clojure, Nodejs, Erlang, Lua, and Haskell should be in your list of workable yet "cool" languages (rather than Ruby or Python).
评论 #5590444 未加载
评论 #5590598 未加载
juiceandjuiceabout 12 years ago
That's some amazing performance out of the JVM.
评论 #5590579 未加载
bimrabout 12 years ago
In order to have an apples to apples comparison, wouldn't you at least need to have an opcode cache for the PHP frameworks?<p>PHP has to live and die upon each request, but Java culls all of its classes together on the first request and from then on it is running from RAM.<p>Many PHP deployments, and most serious ones, will opcode cache in addition to database and page cache. You could rerun these tests with opcode cache only, leaving page and database caches for your "cache" round of tests. That will at least approximate the dramatic differences in architecture between Java and PHP.
评论 #5591058 未加载
mrongeabout 12 years ago
I'd love to see these benchmarks on a beefier ec2 instance type. I'm curious how it would compare to the dedicated hardware
评论 #5591808 未加载
shaydocabout 12 years ago
Do you have any plans to run asp.net MVC webapi on .net 4.5 using the task parallel library ?
deadA1iasabout 12 years ago
Why is ASP.NET never included in these type of benchmarks?
评论 #5590801 未加载
评论 #5590140 未加载
评论 #5590326 未加载
评论 #5590128 未加载
actfabout 12 years ago
These are really great - one additional test I would love to see would be Mono Asp.net MVC. I think it would interesting to see how it compares to the JVM.
babyabout 12 years ago
I just don't understand how could Symfony2 be so fast considering it's pretty huge (twig itself is already huge).<p>I guess it's time I consider switching from CI.
评论 #5590410 未加载
评论 #5590282 未加载
评论 #5590473 未加载
g3raldabout 12 years ago
Wow I'm surprised that Rails has obtained better performance than PHP frameworks, and on many occasions has proved better than DJango...
评论 #5590281 未加载
benhanks040888about 12 years ago
Is the laravel on the benchmark Laravel 3 or Laravel 4? Still, CodeIgniter is the fastest PHP framework? Wow.
评论 #5590321 未加载
nakovetabout 12 years ago
Where can I find this gemini? I went to environment and it's the only one that doesn't have a link.
评论 #5590304 未加载
stcredzeroabout 12 years ago
I'd be more interested in benchmarks that reflect production configurations.
评论 #5590991 未加载
whalesaladabout 12 years ago
A few things that I noticed, just random notes, take 'em with a grain of salt. I'm not running any of the code myself, I'm just browsing around the Github repo. This is a birds-eye analysis. I was a Django developer for a long time and recently went to the dark side and love Rails at the moment.<p>It's important to remember that database connections are a killer in most web apps. This is why a lot of people use connection pooling (in the postgresql world, for example, you have pgbouncer and pgpool). When you can pool and persist connections, you'll have a faster web app (assuming it interfaces with a db, as many of their tests do).<p>Last time I checked, Django does NOT provide built-in connection pooling, whereas Rails does. This could be part of the reason that Rails outperforms in most of the tests. That says something ... but it would be cool to see both frameworks strutting their stuff on an identical connection scenario.<p>Also note that the MySQL connection limit for nodejs is set to 256 connections, whereas Rails set to use a pool of 5. Since these tests are hitting the DB, I'd like to see some more consistency with the way connections are setup. Not that 256 connections is better than 5, but if all those connections are open, it's no wonder that an async framework can hammer the DB so much faster than Ruby chugging along on a pool of 5.<p>In the Node examples, it's important to note the difference between the mysql and mysql-raw tests. And to remember this when comparing to things like Django and Rails which use ORM's like ActiveRecord. In the raw msql test, node is really quick because there is no overhead there for mapping a row to a model.<p>Also, a lot of the viewers of these benchmarks might just look at the big bar for X framework and think, "fuck it, that's my next project". Sure, go for it. But also remember that the type of app you're building comes into play a lot. These tests do not have any state. That's why these bare metal frameworks like Gemini are doing so well. Node does great here because it's async. But certain apps (like your future project) isn't going to fit into the mold of an async hello world test. I'm not knocking the framework tests here, they're awesome, but just realize this.<p>One area where Django outperforms Rails is in the realm of JSON serialization. I'm asusming here that Python's native JSON engine is quicker than Ruby's. I learned this with Rails the hard way. With an out-of-the-box Gemfile, you're gonan have a bad time. In Ruby-land there are a few ways to boost your octane here. You start with MultiJSON and then take your pick of the many JSON engines in the world today. MultiJSON being an adaptor that lets developers put the choice of a JSON engine in your hands. Popular ones are yajl-ruby (ruby bindings for yajl, which is C) and oj. If you're doing JRuby, I hear that Jackson is super effing fast so that's nice. I'd love to see the JSON tests with various JSON parsers (at some point these guys are going to get funding for being the benchmark company, having every framework with every possible configuration on display)<p>Django Protip: read up on select_related and prefetch_related. Similarly, in Rails land, you'll want to learn about eager loading with things like includes() and joins(). This will let you do one or few queries up front, rather than doing a bunch of subqueries inside of a big loop. One big query is usually better than a billion tiny ones.<p>A reason why the PHP apps are getting murdered is because in a typical configuration your application initialized on every single request. Opcode caching can help here since that reduces that boot period. This why in a production environment you can modify a PHP file and instantly see the results. Versus (depending on the application) a Ruby or Python app, where the app is booted initially and until that worker or process is restarted, is running the code it started out with. This is for speed. This is also why, for example, a big Rails project (or better yet, a Java project) has a significantly longer warm-up period. It's loading the entire app. This is a big reason the PHP frameworks are suffering here, since they have a lot of overhead to load for each request. Whereas the raw PHP doesn't have that overhead.<p>PHP protip: Roll with opcode caching (APC is a favorite of mine, it's very simple to install and configure, and provides a handy web gui for viewing various details of things like cache hit so you can optimize it). Also, please throw apache and mod_php away if you can and run with Nginx + php_fpm ... although if you're an HN reader then you're probably already doing this.<p>The JVM is balls-quick because, again, unlike something like PHP, it gets booted up and then does it's thing. The JVM is infamous for it's slow startup process (which really varies based on a billion things) but there is a reason for that: the JIT. The slow startup is due to bytecode generation, I believe. It churns that bytecode into machine code and during that process it can make optimizations like unrolling loops. I'm no Java pro though. I don't like working with the language (the verbosity kills me, and XML is painful), but I have an immense deal of respect for its performance.<p>To be honest I am really going to take a solid look at the Play framework for my next side project. It's got a lot of the pragmatism of Rails but it runs on the JVM and comes in a Scala variant. These are things that I am stoked about dipping my toes into.<p>Finally, my hat is off to the folks running these. Great work!
评论 #5591744 未加载
AlexSerov51about 12 years ago
It would be useful to try the HybridJava framework as well.
g3raldabout 12 years ago
I want to try Gemini. Can you give me some tips to start?
zalewabout 12 years ago
what's "django stripped"?
评论 #5590084 未加载
camusabout 12 years ago
It's a bit strange to include node with mongodb in the tests. Because it changes 2 variables in the test ( the web solution and the database ). You should stick to mysql(or whatever DB you are using) with every tests or it doesnt sound that serious.