I've been kicking around an idea like this for a while. The train of thought that brought me there was the recognition of a distinct memory hierarchy in today's common distributed applications that parallels the one in your computer.<p>So your computer has memory banks that (as a rough first approximation) each get ~10x bigger, but with ~10x greater latency. The neat part though is that global consistency for writes happens with L1 and L2 working together, so pretty damn high up in the hierarchy. In contrast distributed applications typically at best will have a write through cache where global consistency happens all the day down at the actual data store. Exploring the idea of distributed MOESI where one client, because they had the permissions to write something in the first place, can then be the owner for that database row as it's still being flushed out seems like a great basis for a distributed system that might not even need the dedicated datastore at all anymore, but a sea of clients participating in coherency and replication. Albeit this has oodles of consistency and availability problems that very well might kill the whole concept, like how MOESI would absolutely fall over when trying to hotplug CPUs at arbitrary times.
Love this. We’re building a similar architecture at Splitgraph [0] which we refer to as a “Data Delivery Network.” The basic idea is we implement a proxy that forwards queries to backend data (either live data sources via an FDW, or versioned snapshots via our Docker-inspired “layered querying”).<p>Soon anyone will be able to connect data sources by adding their read only credentials in the web (already possible via CLI but undocumented). The idea is to make exposing a database to the DDN as simple as exposing a website to a CDN.<p>We’ve designed all this to be multi-tenant and horizontally scalable, but we’re not actually running it on a distributed network yet. Personally, I’ve followed Fly for a long time and always loved the angle you’re taking. If any of you at Fly read this and want to potentially collaborate on a solution in this space, my email is in my profile.<p>[0] <a href="https://www.splitgraph.com" rel="nofollow">https://www.splitgraph.com</a>
The way this is implemented, with the ability for an application server attached to a replica to say "error: this needs to perform a write - hey CDN, replay this request against the region with the database leader in it" is SO clever.
I first read the title as "Globally Distributed Progress" and I was like "if an article with this title made it to the front page of HN it must sure be some interesting read.<p>I am now slightly disappointed.
Running a master in us-west and having read replica in europe, what could go wrong ...<p>My advice is don't do those things in the blog and keep the DB and the app in the same region it will save you so many problems. If you can't just use a DB that was designed for that like Spanner or Cockroach.
It reminds me of Yugabyte, which is Postgres compatible<p>> YugabyteDB is the open source, distributed SQL database for global, internet- scale applications with low query latency, extreme resilience against failures. *<p>However Fly is nicer from a developer's point of view, because it doesn't require you to learn a new query language, you can write good old Postgres.<p>* <a href="https://www.yugabyte.com/" rel="nofollow">https://www.yugabyte.com/</a>
> It is much, much faster to ship the whole HTTP request where it needs to be than the move the database away from an app instance.<p>Does that mean that the app server would just fail, and then your network replays the same HTTP request (made by the client to the edge), toward an instance running the app server inside the region where there the Postgres primary instance is running? If so, you should then be able to figure out which client request provoked the SQL error and the app server would report some errors, right? Otherwise, the app server should be able to tell you if it failed because the Postgres server does not support write statements, which would require some minor adjustment in the app server code, isn't it?<p>I am sorry if I missed something from your description, but I am trying to figure out what the flow would look like. Thank you!
This is a nice solution, although I prefer for my application to be aware of read vs. read-write connections.<p>Question about fly.io - how strict is the docker image requirement? How easy would it be to deploy a go application + systemd service?
I know this is hacker news and all and you’re trying to be fun, but as I’m looking at platforms like fly for a future saas, the first sentence (cool hack) rubbed me the wrong way. Would rather hear “we found an elegant design” and are working hard to make it bulletproof… etc.<p>Databases are like that, the place to get serious.<p>Sure I won’t let it deter me from a proper eval, but I could definitely see it scaring away suits.
I tried this a few months ago. Many features were missing then. I couldn't set up backups, and connect directly to `psql`. It was such a pain. I don't know if they've addressed those issues yet. It'd be a hard sell for me to store a database system for a production app in a Docker image.
Also relevant: <a href="https://fly.io/docs/getting-started/multi-region-databases/" rel="nofollow">https://fly.io/docs/getting-started/multi-region-databases/</a>
How are POST requests replayed into different regions? Does your edge proxy hold the whole POST requests (body and all?) just in case it needs to be replayed?
Oh look a distributed database with nary a mention of CAP.<p>And they are allowing multi-master writes? What is the update collision resolution? cell timestamps? vector clocks?