Well, the author cares enough about RethinkDB to test it, even if he's a mongodb fan, even if his first benchmark was wrong, he was right to publish it: you all helped him when you pinpointed the problems in his tests... Thanks you for that.<p>I don't see any marketing here, just the "do your own benchmark" best practice, and the "share with community" best practice... Does it make it a perfect benchmark? No, but at least he tried... and the author has corrected the discrepancies since then.<p>Now imagine the benchmark was against [your favorite DB here] with even stronger results against RethinkDB. Notice how the most upvoted comment is joking about MongoDB. The second one is a pro-mysql comment. What's the point? Would it have been a better benchmark if it read "mysql is 10x faster than RethinkDB?" or "MongoDB is even slower than RethinkDB"?
This old comic piece seems to still apply: <a href="http://www.mongodb-is-web-scale.com/" rel="nofollow">http://www.mongodb-is-web-scale.com/</a>
From the linked benchmark:<p>> In RethinDB, you have to create databases and tables manually and it will raise an exception if they already exist. Compared to MongoDB that could be an inconvenience for some(and me) - one of the things I find appealing in MongoDB is the fluid interaction with databases<p>... well at least now I don't feel so bad about having some old MySQL stuff still in production. MySQL already has too much "fluidity" in dealing with my data...
> I expect to be able to click on a table and see the rows inside - like all the tools out there for Mongo, MySQL, PostgreSQL, etc.<p>He does have a point there. It is slightly annoying having to type out a query when I just want to browse the data. (No real biggie though..)
This is why a lot of commercial databases have a "you may not publish benchmarks" clause in the license.<p>It seems unfair and restrictive, but benchmarking is <i>hard</i>, and even where users get everything else right every data load is different.<p>It's easy to see why companies don't want writeups like this one dominating searches for "<database> performance".
I've been working with RethinkDB recently on some slightly unusual things and the Rethink team has been first class. They've got great support on IRC and GitHub and are open and friendly. I highly recommend them.
Any output like this, unless maliciously fallacious, is contributing in some way to the general understanding of the software concerned and benchmarking best-practices, even through its mistakes.<p>It's the job of the reader to judge their sources wisely, and interrogate what they read, rather than the job of the author to conduct their explorations in private.<p>Understandably, it can be frustrating for people involved in the projects but that's just the nature of the beast. They can do things to help their cause by championing good examples of benchmarking, even those which don't look upon them favourably.
Benchmarking is hard and a lot of reports are bogus. However they are still very useful for a lot of developers.<p>Benchmarking programming languages got better. E.g.: <a href="http://benchmarksgame.alioth.debian.org/" rel="nofollow">http://benchmarksgame.alioth.debian.org/</a> gives roughly idea about performance of programing languages.<p>I wish something similar existed for databases. I think exact figures would be hard to get, but I believe there are many 2x 10x differences that we should be aware of.
Not sure what's worse here - people relying on third party benchmarks (hint: always do your own; see how a tool performs on your data, on your hardware, for your problem set), or the fanboy-ish panic when they are unsettled that a benchmark might make their chosen toy less shiny?