MongoGate — or let's have a serious NoSQL discussion

My big concern with so-called NoSQL solutions is the "culture" that seems to be brewing there.<p>If you go to the "Don't use MongoDB" post ( <a href="http://news.ycombinator.com/item?id=3202081" rel="nofollow">http://news.ycombinator.com/item?id=3202081</a> ) you will read some, IMO, extremely worrying comments from a few pro-NoSQL users including antirez (Redis).<p>For some reason NoSQL now apparently means "unreliable datastore for unimportant, throwaway data" and defaults are chosen accordingly. Why the hell is that?<p>NoSQL for me doesn't imply anything other than "no SQL", and at a stretch "no schema" - this makes a lot of sense for many of us who routinely need to create databases that are logically trivial. In many cases they are a bunch of glorified persistent hash tables that usually don't fit in memory. But this doesn't mean they aren't critical. Why would it have to? This isn't anything new either, we've had Berkeley DB for a long while. It's just a bit of the dry side and it may fall short in many cases.<p>What I was looking forward to and I hoped I could find in the "NoSQL scene" is an alternative to traditional DBs but without the overhead that many times is not necessary (but sometimes is, and I intend to continue using PostgreSQL when appropriate). Ideally, something as simple as mongoDB appears to be (tried the interactive tutorial).<p>So when exactly NoSQL stopped meaning "no SQL" and started meaning "unreliable cache"? Other than the simplicity, I fail to see where it would fit in the market then (other than the amateur market). There are better, stablished DB caching solutions. There are persistence libraries in any moderately language. There are reliable databases that are fast enough when you have the budget to scale to several dedicated servers.<p>How about Riak?

Looking at HN today - it's full of hate and negativity.<p>Kudos to the developers that rise above this, often working for nothing, to build the awesome tools that future generations will use to build awesome apps.

I'm calling troll.<p>Having a serious discussion about NoSQL databases begs the exact same question as having a serious discussion about cancer: what kind would you like to have a serious discussion about?<p>I think the most important lesson we can learn from NoSQL in general is that the idea of a one-size-fits-all database is becoming dated. NoSQL databases certainly <i>don't</i> solve the problems the author points out, and they probably never will. In fact that's the point. By not solving one set of problems, you allow yourself to solve another set of problems.<p>How about we use databases to solve the problems they were meant to solve, rather than basing our choices on whatever the popular opinion is at the moment.

Hi zeit_geist,<p>to me the problems you've described in your blog post are specifically application model problems. I think we shouldn't abstract the application model into the database, but the database into the application model.<p>I know a very innovative French developer who wrote an application server, that comes integrated with the database. In this very way you just call the exported functions provided by your database directly. How you model your (re)caching/(re)indexing and other application needs is totally up to you. This a) a freedom you barely find anywhere else. b) bare to the metal development of an application c) the most effcient way to develop an application. (b/c you only implement what you need and don't use a generalized construct that serves a general purpose very well, but doesn't scale with your application very well)<p>I would recommend to implement an application using the pattern that you know works best for the application, if you don't know it yet, then it's time to read books that enlighten our horizon of available solutions until we can start developing again.<p>I will show you an example of what I mean.<p><a href="http://gwan.ch/api#kv" rel="nofollow">http://gwan.ch/api#kv</a><p>This is how I think is the most elegant way to interact with a(n integrated) database.<p>I am curious on what you think about this. I know I've not referred to the points in your post, but I've read it carefully. Thanks for hearing me out. I'm sorry I didn't post to your blog, but I prefer to post without subscribing to an external party. You limit the users who can answer this way imho. I'm not sure if it helps you to keep out trolls/spammers, but it sure helps to keep response rate low.

"The problem is: NoSQL is not a solution at all. It's a trade-off."<p>Bingo, that is <i>precisely</i> what NotOnlySQL is all about. For example you trade some consistency guarantees for the ability to scale out.<p>Uninformed (has the author heard about the CAP theorem?), either-or diatribes like this article don't really serve any purpose other than sowing discord.<p>This is just like "C++ is better than Java is better than..." type flames wars. :)<p>We use Oracle and we use HBase. We would never replace Oracle with HBase for all of our data needs. At the same time we have need for a store that scales beyond what even Oracle can provide (and yes, we use RAC with multi TB caches across a database instance).<p>For the same reason we use Java, C++, Scala, Perl, Closure, Bash, JavaScript, etc... The right tool for the right job.<p>Personally what I would like to see is:<p>* secondary indexes<p>* snapshot isolation (in leu of global transactions, which will never scale).<p>Disclaimer: HBase committer here.

I find one of his counterexamples amusing:<p><i>Managing Highly-dimensional data and access to it: ...I'm thinking of e.g. geo/spatial data here. Where are the solutions out there?</i><p><a href="http://www.mongodb.org/display/DOCS/Geospatial+Indexing" rel="nofollow">http://www.mongodb.org/display/DOCS/Geospatial+Indexing</a>

The original document to which you refer (that I refuse to refer to as the "MongoGate" document :-) is about system reliability. Those types of problems can exist in any database system and are not specific to every NoSQL database system. The document claims that MongoDB doesn't perform well under very high loads in a replicated environment.<p>Yes, NoSQL doesn't fit the problem you're trying to solve. Perhaps there are a set of problems that are difficult to solve with NoSQL, but there exists sets of problems for which NoSQL databases are perfectly suited. So, I would modify your post to state that NoSQL isn't the solution to every problem, but don't think you're uncovering some big secret, because most people already know that.

I just recently published an article describing my experience migrating from SQL Server to MongoDB, you can read it here: <a href="http://www.wireclub.com/development/TqnkQwQ8CxUYTVT90/read" rel="nofollow">http://www.wireclub.com/development/TqnkQwQ8CxUYTVT90/read</a><p>I tried my best to describe both what we gained and what we lost after the transition. At the end of the day, MongoDB (and other NoSQL solutions) are different tools for different jobs. Obviously it takes investment to master a new tool and we almost aborted the migration in two different occasions simply because we didn't know enough about maximizing MongoDB performance. Now that the dust has settled and with all things considered, I am glad we didn't.

Disclosure: I wrote a product called Citrusleaf, which also plays in the NoSQL space.<p>I also want a better discussion of NoSQL. It isn't fair to hate on databases without understanding the pressures of operations. I saw a friend's company where a big, fancy oracle system lost <i>all</i> of its data on their main test/dev system at a crucial moment - lost over 100,000 user accounts, including those of executives of key customers. They were forces to merge with a competitor about 4 months later.<p>You need to take database backups, you need to stage your systems. You need to have extra hardware on hand.<p>Some of our customers at Citrusleaf continue to "run with scissors". I like the attitude, but we've had to talk sternly with them about the benefits of staging, bucket testing new releases (app and db), and penciling out the realistic hardware requirements.<p>The new crop of distributed databases provide an immense opportunity for all of us. We can write more agile applications than ever before, and as a community we all need to understand the <i>benefits</i> of flexibility. This includes your entire organization.<p>That being said, there are technology differences between the NoSQL solutions, and at Citrusleaf we've focused on operations and deployability. My co-founder ran Yahoo Mobile's engineering and ops group, so understands the tradeoffs. We have a group in India (hi guys!) of great developers (not support guys) simply to make sure that when you've got a problem at 3am there's someone to take care of you.<p>Performance is important in this agile world, and Citrusleaf has it. <a href="http://bit.ly/rRlq9V" rel="nofollow">http://bit.ly/rRlq9V</a><p>A slide I showed at HPTS (the high performance transaction systems conference) showed a Zynga game on the right, and an EA facebook game on the right. Zynga is an amazing machine in terms of getting huge, rich applications to market. Every pixel is covered with things to do, artwork, everything. And they're rolling out new games every week, and I haven't ever seen downtime (unlike Netflix Streaming, which has maintenance on a regular basis).<p>Zynga has been a huge proponent of NoSQL (but not Mongo) since its inception, and although I don't know what EA does internally (maybe they use the same tech but have other agility issues), NoSQL is clearly part of a high scale, rich application need.<p>Join or be flattened.