This is very cool!<p>FoundationDB excites a lot of people because it's an extremely scalable and extremely reliable distributed database that supports ACID transactions, and which is both open-source and has Apple standing behind it. And yeah, all of that is pretty nice.<p>But arguably the real power comes from the fact that it exposes a relatively low-level data model that can then be wrapped in one or more stateless "layers". All of these layers write to the same storage substrate, so you can have your document database, your SQL database, your time-series database, your consensus/coordination store, your distributed task queue, etc., etc., but you're only actually operating one stateful system. Your SREs will thank you.<p>Writing these layers to be scalable and high-performance can be challenging, but it looks like Apple is actively doing it and willing to release the results to the rest of us. This also suggests that their previous open-sourcing of the MongoDB-compatible document layer wasn't a one-off fluke. All of this is very good news for everybody who needs to run databases in the real world.<p>Full disclosure: I worked on FoundationDB a long, long time ago.
Apple low key does some cool server projects with a Java bent. They've contributed to Netty (well, they hired core developers).[1]<p>They've been basically put them to work reimplementing it in Swift.[2] It's open and out there but not a lot of people paying attention. While it's still early days I think there may a year where, suddenly, Swift on the server is a super serious thing and all this work they've been doing on little old CloudKit kind of takes over the world.<p>Just a fun prediction.. but it wouldn't be the first time Apple pulled something like that.<p>I do like that Swift's non-tracing garbage collection model is well suited for server apps. Rust is cool too but maybe Swift would be a little friendlier and thus better suited to inherit Java's mantle. I mean can you just imagine if Apple is slowly building up Swift to overtake Java on the server? That that's one of their long game master plans? I know that sounds completely crazy.. It just might work. They do run one of the biggest data center networks in the world so they have a pretty good testbed and can justify a hefty R&D budget.<p>[1] <a href="https://www.infoq.com/presentations/apple-netty" rel="nofollow">https://www.infoq.com/presentations/apple-netty</a><p>[2] <a href="https://github.com/apple/swift-nio" rel="nofollow">https://github.com/apple/swift-nio</a>
I had built a layer like this one for my startup Bagcheck called Havrobase[1] (it was on top of HBase/Solr, here is the motivating blog post[2]) that ultimately I put on top of MySQL/Solr and other stores. Later, when we started Wavefront, I ported that layer to FDB and that still powers their metadata. Really a good fit and very much like this record layer. I highly recommend this approach for 24/7 services as you never need to have maintainence windows for schema upgrades and the like.<p>[1] <a href="https://github.com/spullara/havrobase" rel="nofollow">https://github.com/spullara/havrobase</a>
[2] <a href="https://javarants.com/havrobase-a-searchable-evolvable-entity-store-on-top-of-hbase-and-solr-d305f90a3eaa" rel="nofollow">https://javarants.com/havrobase-a-searchable-evolvable-entit...</a><p>Initially at Wavefront we were using HBase for telemetry, Zookeeper for the service mesh and MySQL Cluster for entity metadata. All that was moved on top of FDB with 3 different layers that we developed.<p>I'm excited that this kind of database is now going to be available more broadly and with the confidence that CloudKit is using the same technology since to date implementing something like this was basically a DIY project.
Glad Apple is releasing all of this, I wonder what kickstarted it all?<p>The paper is rather interesting: <a href="https://www.foundationdb.org/files/record-layer-paper.pdf" rel="nofollow">https://www.foundationdb.org/files/record-layer-paper.pdf</a>
This might be the first good alternative to etcd for configuration stores that need real-time updates.<p>Like Kubernetes.<p>Many Kubernetes scaling issues are etcd-related.<p>RethinkDB is dead-ish, and CockroachDB is treating their changefeeds as an enterprise feature that requires a Kafka instance to stream to :(
Congrats to the team at Apple for getting this released! They have had a busy few months with getting the document layer released, the FDB Summit, and now the record layer.
Very interesting. I've been looking closely at FoundationDB as a way forward (to replace RethinkDB and Cassandra in existing systems). It's one of the few contenders for a really interesting take on a distributed database.<p>I am not sure if I will use the record layer (I've been planning to write "my layer" myself), but it will definitely be an interesting thing to look at.
I learned that basically all of Imessages and contacts are stored on foundation DB, it's pretty great this is making it into opensource. Thanks Apple!
Seeing this soon after the AWS “wire compatible with Mongo” kerfuffle, it makes me think: it would be amazing if the cloud vendors would offer a managed FDB service. An open-source, cloud-agnostic, horizontally scalable, document-oriented transactional database would be an incredible tool. I know AWS is going in the opposite direction these days with proprietary “wire compatible” services but a guy can dream...
Few doubts:<p>1. Any reason to write it in Java instead of C, C++, Rust, etc?<p>2. Any reason to use Protobuf instead of Flatbuffers, Avro, etc?<p>3. Can FoundationdDB be used with Apache Arrow?
Has anyone ever used FoundationDB and <i>not</i> found it successful? All I read is "it supports RDMS + NoSQL and can be distributed". So what use cases <i>doesn't</i> it solve?
So why would Apple be doing this now? Maybe preparing the terrain to enter the cloud space and compete with Azure and AWS in a couple of years?<p>After all, it's no mystery Apple wants to expand their services revenue. Their hardware revenue it's not growing as much as it used to.
The preprint of the paper is now up on arXiv.org: <a href="https://arxiv.org/abs/1901.04452" rel="nofollow">https://arxiv.org/abs/1901.04452</a>
Does that mean FDB now supports secondary indexes?<p>If that's the case, how does FDB compare to ScyllaDB now that they both have secondary indexes?
Can someone please ELI5/executive summary to me what are the benefits of FoundationDB? Assuming I know the basics of PostgreSQL and ElasticSearch? I see some hype around it, but I can't understand what's the breakthrough. As a helping question: can you maybe try to tell me who are the expected users of it, vs. PSQL, ES? Or, when I should choose it over them? Also, what are its disadvantages? (I suspect bigger complexity, and bigger cost/worse effectiveness at small scale?) TIA!