SenseiDB: Open-source, distributed, realtime, semi-structured database

I can't speak for SenseiDB itself, but I think that LinkedIn deserves a lot of credit for the technology they have been open sourcing in the last few months. Fantastic to see.

Looks like the marketing copy is off from the actual implementation, AFAICT.<p><pre><code> Sensei (先生) means teacher or professor in Japanese(http://en.wikipedia.org/wiki/Sensei). It shares the same pronunciation and writing with the Chinese word that has the same meaning. This name indicates that the system can be used in place of Oracle database in many applications. </code></pre> Okay, so on the name alone I can replace my Oracle database! Great!<p>Seriously though it goes on...<p>They claim the database is ACID: <a href="http://javasoze.github.com/sensei/data-guarantee.html" rel="nofollow">http://javasoze.github.com/sensei/data-guarantee.html</a><p>But they built the entire thing around "eventual consistency."<p>And statements like:<p><pre><code> "Sensei provides a high-level of durability by maintaining N replicas of each shard to guarantee a level of availability and fault-tolerance" </code></pre> Don't seem to make sense when talking about ACID given that a <i>write</i> operation will happen <i>at some point</i>. Looks like the data event producers will shard the data across N replicas without quorum... so there's no guarantee that there will be N replicas available... is that right (and that the transaction won't be lost mid-stream either)?<p>Skimming through the source it doesn't seem to be doing anything terribly revolutionary... and I can see the usefulness of the trade-offs they made in this database for certain scenarios. However I don't think the claims of ACID guarantees and "real time" are particularly representative of what this DB will actually do. They just don't seem to jive with "eventual consistency" models.<p>I'm not a hardcore database guru though so maybe I'm missing something?

Looks like someone non-LinkedIn posted this so the rest of the team is unlikely to be around for a few more hours. Let me know if you have any questions! I worked mostly on the distribution &#38; rpc piece.

3 separate NoSQL stores from ONE company? And their primary business is not Databases? There are currently more than 100 different NoSQL solutions - I believe we will see a consolidation of NoSQL in 2012

SenseiDB looks really impressive. We were actually thinking about the same problem when we built ThriftDB (<a href="http://www.thriftdb.com" rel="nofollow">http://www.thriftdb.com</a>) for Octopart so it's especially interesting to see LinkedIn's implementation.<p>Looking through the documentation a couple of things come to mind:<p>* Does SenseiDB support nested data structures?<p>* What happens when you modify the schema? Can you add/delete/modify attributes?

How many home-grown databases have linkedin developed in the past few years?

Options are great (and thanks for contributing to the OS community), but the biggest barrier to adoption will be confusion about all the NoSQL solutions floating around right now (including two others from LinkedIn). Therefore, I recommend that you spend more time on the homepage comparing Sensei with other NoSQL stores. Perhaps take a two-tier approach for those coming from Oracle or MySQL: why NoSQL, and if NoSQL, why Sensei NoSQL?

Looking at its config, I'm reminded of Cassandra 0.6, but then I see the ZooKeeper functionality. Interesting. Love the open-sourcing! I showed an example of how to distribute inter-mingled DBs using MQtt a bit ago, but never posted here: <a href="http://franklovecchio.tumblr.com/post/13051814890/mqdb-distributing-databases-with-mqtt" rel="nofollow">http://franklovecchio.tumblr.com/post/13051814890/mqdb-distr...</a>.<p>One thing is for sure -- NoSQL isn't so mysterious anymore :)

So to get data into this database, I have to have a Gateway implementation in Java?<p>Would I have to publish messages to Kafka in my language of choice to avoid writing any Java?

From a quick glance at the docs there's no database-side aggregations. Is that correct?

Can Sensei be used as a replacement to MySQL + lucene?

This is awesome.<p>Thanks for sharing this with the world. :)

Is it single-point-of-failure free?

A truly amazing peace of technology

Nice looking wheel. I have a feeling it could be rounder though.