NYTimes.com Announces Map/Reduce Toolkit

I have to give a lot of respect to NYT. Whether this project sticks or not, it's things like this that make me think NYT is going to be one of the few newspapers to survive the crisis hitting the papers (albeit in a much smaller and much different form). They're one of the few newspapers at least <i>trying</i> to get it. Others are just complaining while hemorrhaging money.

Interesting that it's built in Ruby. I was under the impression that NYTimes did pretty much all of their dynamic language work in Perl.

Anyone know why they built this rather than use Hive or Pig? One thing that drives me nuts is that all of these MR tools are very slow because they don't take advantage of indexes and use inefficient storage (e.g. in this case, plain text files), both of which would likely improve query performance considerably.

I think it's interesting that this was released by the New York Times. . .this could be prove to be an interesting new model/trend for newspaper publishers to remain as a viable competitor in the 21st century, given the bad-rap they seem to be giving themselves these days as news<i>paper</i> publishers. D=

This is pretty meaningless -- there's already a THRIFT interface which allows easy job creation and control as well as HadoopStreaming which allows access to creating map-reduce jobs for anything using stdin/stdout.<p>This has dubious benefit, and just adds another unnecessary layer into this process. I'm not sure why this is news.

last.fm did something similar, called Dumbo, for writing your Hadoop jobs in Python.

the least they could do is post a link to the code on github to help out a startup -- why use google code?