LiteFS

This is distributed SQLite 3, running (I assume at least partially managed?) LiteFS[5] for you. Which is pretty cool!<p>What I&#x27;d like to have seen is how this compares to things like rqlite[1] or Cloudflare&#x27;s D1[2] addressed directly in the article<p>That said, I think this is pretty good for things like read replica&#x27;s. I know the sales pitch here is as a full database, and I don&#x27;t disagree with it, and if I was starting from scratch today and could use this, I totally would give it a try and benchmark &#x2F; test accordingly, however I can&#x27;t speak to that use case directly.<p>What I find however and what I can speak to, is that most workloads already have database of some kind setup, typically not SQLite as their main database (MySQL or PostgreSQL seem most common). This is a great way to make very - insanely, really - fast read replica&#x27;s across regions of your data. You can use an independent raft[3][4] implementation to do this on write. If your database supports it, you can even trigger a replication directly from a write to the database itself (I think Aurora has this ability, and I think - don&#x27;t quote me! - PostgreSQL can do this natively via an extension to kick off a background job)<p>To that point, in my experience one thing SQLite is actually really good at is storing JSON blobs. I have successfully used it for replicating JSON representations of read only data in the past to great success, cutting down on read times significantly for APIs as the data is &quot;pre-baked&quot; and the lightweight nature of SQLite allows you to - if you wanted to naively do this - just spawn a new database for each customer and transform their data accordingly ahead of time. Its like AOT compilation for your data.<p>if you want to avoid some complexity with sharding (you can&#x27;t always avoid it outright, but this can help cap its complexity) this approach helps enormously in my experience. Do try before you buy!<p>EDIT: Looks like its running LiteFS[5] not LiteStream[0]. This is my error of understanding.<p>[0]: <a href="https:&#x2F;&#x2F;litestream.io&#x2F;" rel="nofollow">https:&#x2F;&#x2F;litestream.io&#x2F;</a><p>[1]: <a href="https:&#x2F;&#x2F;github.com&#x2F;rqlite&#x2F;rqlite" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;rqlite&#x2F;rqlite</a><p>[2]: <a href="https:&#x2F;&#x2F;blog.cloudflare.com&#x2F;introducing-d1&#x2F;" rel="nofollow">https:&#x2F;&#x2F;blog.cloudflare.com&#x2F;introducing-d1&#x2F;</a><p>[3]: <a href="https:&#x2F;&#x2F;raft.github.io&#x2F;" rel="nofollow">https:&#x2F;&#x2F;raft.github.io&#x2F;</a><p>[4]: <a href="https:&#x2F;&#x2F;raft.github.io&#x2F;#implementations" rel="nofollow">https:&#x2F;&#x2F;raft.github.io&#x2F;#implementations</a><p>[5]: <a href="https:&#x2F;&#x2F;github.com&#x2F;superfly&#x2F;litefs" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;superfly&#x2F;litefs</a>

This is really cool! Unfortunately i primarily am interested in offline databases so perhaps i&#x27;m just not the target audience. However i have to ask, on that note, does this have any application in the offline space?<p>Ie i wonder if there&#x27;s a way to can write your applications such that they have less&#x2F;minimal contention, and then allow the databases to merge when back online? Of course, what happens when there inevitably _is_ contention? etc<p>Not sure that idea would have a benefit over many SQLite DBs with userland schemas mirroring CRDT principles though. But a boy can dream.<p>Regardless, very cool work being done here.

&gt; Developing against a relational database requires devs to watch out for &quot;N+1&quot; query patterns, where a query leads to a loop that leads to more queries. N+1 queries against Postgres and MySQL can be lethal to performance. Not so much for SQLite.<p>This is misleading AFAICT. The article(s) is actually comparing remote RDBMS to local RDBMS, not Postgres to SQLite.<p>Postgres can also be served over a UNIX socket, removing the individual query overhead due to TCP roundtrip.<p>SQLite is a great technology, but keep in mind that you can also deploy Postgres right next to your app as well. If your app is something like a company backend that could evolve a lot and benefit from Postgres&#x27;s advanced features, this may be the right choice.

This approach is very appealing to me :) curious about how people handle schema migrations when using this approach.<p>I segment sqlite files (databases) that have the same schema into the same folder. I haven&#x27;t really had a case where migrations was really a concern, but I could see it happening soon.<p>Seems like in my deployment, I&#x27;m going to need an approach to loop over dbs to apply this change... I currently have a step of app deployment that attempts to apply migrations... but it is more simplistic because the primary RDBMS (postgresql) just appears to the application as a single entity which is the normative use-case for db-migrate-runners.

&gt; To improve latency, we&#x27;re aiming at a scale-out model that works similarly to Fly Postgres. That&#x27;s to say: writes get forwarded to the primary and all read requests get served from their local copies.<p>How can you ensure that a client that just performed a forwarded write will be able to read that back on their local replica on subsequent reads?

10 years ago fly.io is the company I wanted to build. Something with massive technical depth that becomes a developer product. They&#x27;re doing an incredible job and part of that comes down to how they evangelise the product outside of all the technical hackery. This requires so much continued effort. AND THEN to actually run a business on top of all that. Kudos to you guys. I struggled so much with this. Wish you nothing but continued success.

I wonder if using FUSE has had any appreciable impact on performance, particularly read performance. I ask because FUSE has historically had a reputation for being slow, e.g. with the old FUSE port of ZFS.

The pain is that this approach is suitable on VPS&#x2F;IaaS where disk volume is supported. As a solo dev, I only use PaaS kind of infra, there are just a few PaaS i&#x27;m aware of that support attachable disk. Fly, Render, .. nothing else?

Where is the data actually being stored in this setup? A copy on each machine running the application? If so, is there another copy somewhere else (e.g. S3) in case all nodes go down?<p>Also, what happens if the Consul instance goes down?<p>If my application nodes can&#x27;t be ephemeral then this seems like it would be harder to operate than Postgres or MySQL in practice. If it completely abstracts that away somehow then I suppose that&#x27;d be pretty cool.<p>Currently finding it hard to get on board with the idea that adding a distributed system here actually makes things simpler.

&gt; Second, your application can only serve requests from that one server. If you fired up your server in Dallas then that&#x27;ll be snappy for Texans. But your users in Chennai will be cursing your sluggish response times since there&#x27;s a 250ms ping time between Texas &amp; India.<p>&gt; To improve availability, it uses leases to determine the primary node in your cluster. By default, it uses Hashicorp&#x27;s Consul.<p>Having a satellite office become leader of a cluster is one of the classic blunders in distributed computing.<p>There are variants of Raft where you can have quorum members that won&#x27;t nominate themselves for election, but out of the box this is a bad plan.<p>If you have a Dallas, Chennai, Chicago, and Cleveland office and Dallas goes dark (ie, the tunnel gets fucked up for the fifth time this year), you want Chicago to become the leader, Cleveland if you&#x27;re desperate. But if Chennai gets elected then everyone has a bad time, including Dallas when it comes back online.

I can imagine a database which tries to solve both of these domains.<p>A centralized database handles consistency, and vends data closures to distributed applications for in-process querying (and those closures reconcile via something like CRDT back to the core db).<p>Does this exist?

I actually gave fly.io a whirl over the weekend. was not fun. spent a lot of time on the forums and its pretty clear it has some way to go before it can give AWS or Linode a run for their money.<p>For instance, we run kubernetes on multiple VPS providers, including public clouds with serverless onramp&#x2F;offramps deployed on edge location. Anything under 15 minutes are processed by serverless. Anything longer is offloaded to one of the VPS containers available in every part of the world.<p>I have some more feedbacks ready if you are interested, its a neat idea but not exactly as seamless and easy as the idea proposed since public clouds already offer a way to do this.

Currently, I am running multiple application servers (and lambda functions) using AWS Fargate that access sqlite files (databases) on an EFS share. So far so good, although my use cases are fairly simple.

Well that was fast... [1]<p>Are the readable replicas supposed to be long-lived (as in, I don&#x27;t know, hours)? Or does consul happily converge even with ephemeral instances coming and going every few minutes (thinking of something like Cloud Run and the like, not sure if Fly works the same way)? And do they need to make a copy of the entire DB when they &quot;boot&quot; or do they stream pages in on demand?<p>[1] <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=32240230" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=32240230</a>

Sound like a drop in solution to add high availability to WunderBase (<a href="https:&#x2F;&#x2F;github.com&#x2F;wundergraph&#x2F;wunderbase" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;wundergraph&#x2F;wunderbase</a>). Can we combine LiteFS with Litestream for Backups, or how would you do HA + Backups together?

Seems neat, until you try to do schema migrations. Unless they can guarantee that all containers’ SQLite instances have the same scheme without locking I’m not sure how doesn’t run into the same issues as many NoSQL.<p>CouchDB had this same issue with its database per user model and eventually consistent writes.

Unrelated, but why does the map on their homepage show a region in Cuba? That must be wrong.

ELI5?

Does anyone else bump into the issue, that the fly.io website does not load if requested via IPv6 on Mac? I tried Safari, Chrome and curl and neither work:<p><pre><code> $ curl -v https:&#x2F;&#x2F;fly.io&#x2F;blog&#x2F;introducing-litefs&#x2F; * Trying 2a09:8280:1::a:791:443... * Connected to fly.io (2a09:8280:1::a:791) port 443 (#0) * ALPN, offering h2 * ALPN, offering http&#x2F;1.1 * successfully set certificate verify locations: * CAfile: &#x2F;etc&#x2F;ssl&#x2F;cert.pem * CApath: none * (304) (OUT), TLS handshake, Client hello (1): curl: (35) error:02FFF036:system library:func(4095):Connection reset by peer </code></pre> Requesting via ipv4 works<p><pre><code> $ curl -4v https:&#x2F;&#x2F;fly.io&#x2F;blog&#x2F;introducing-litefs&#x2F; * Trying 37.16.18.81:443... * Connected to fly.io (37.16.18.81) port 443 (#0) * ALPN, offering h2 * ALPN, offering http&#x2F;1.1 * successfully set certificate verify locations: * CAfile: &#x2F;etc&#x2F;ssl&#x2F;cert.pem * CApath: none * (304) (OUT), TLS handshake, Client hello (1): * (304) (IN), TLS handshake, Server hello (2): * (304) (IN), TLS handshake, Unknown (8): * (304) (IN), TLS handshake, Certificate (11): * (304) (IN), TLS handshake, CERT verify (15): * (304) (IN), TLS handshake, Finished (20): * (304) (OUT), TLS handshake, Finished (20): * SSL connection using TLSv1.3 &#x2F; AEAD-CHACHA20-POLY1305-SHA256 * ALPN, server accepted to use h2 * Server certificate: * subject: CN=fly.io * start date: Jul 25 11:20:01 2022 GMT * expire date: Oct 23 11:20:00 2022 GMT * subjectAltName: host &quot;fly.io&quot; matched cert&#x27;s &quot;fly.io&quot; * issuer: C=US; O=Let&#x27;s Encrypt; CN=R3 * SSL certificate verify ok. * Using HTTP2, server supports multiplexing * Connection state changed (HTTP&#x2F;2 confirmed) * Copying HTTP&#x2F;2 data in stream buffer to connection buffer after upgrade: len=0 * Using Stream ID: 1 (easy handle 0x135011c00) &gt; GET &#x2F;blog&#x2F;introducing-litefs&#x2F; HTTP&#x2F;2 &gt; Host: fly.io &gt; user-agent: curl&#x2F;7.79.1 &gt; accept: *&#x2F;* &gt; * Connection state changed (MAX_CONCURRENT_STREAMS == 32)! &lt; HTTP&#x2F;2 200 &lt; accept-ranges: bytes &lt; cache-control: max-age=0, private, must-revalidate &lt; content-type: text&#x2F;html &lt; date: Wed, 21 Sep 2022 16:50:16 GMT &lt; etag: &quot;632b20f0-1bdc1&quot; &lt; fly-request-id: 01GDGFA3RPZPRDV9M3AQ3159ZK-fra &lt; last-modified: Wed, 21 Sep 2022 14:34:24 GMT &lt; server: Fly&#x2F;51ee4ef9 (2022-09-20) &lt; via: 1.1 fly.io, 2 fly.io &lt; &lt;!doctype html&gt; ...</code></pre>

still wondering why projects like <a href="https:&#x2F;&#x2F;www.reactivated.io&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.reactivated.io&#x2F;</a> are using fly.io

My dream would be if this supported geo-partitioning. In my field people are pretty sensitive about GDPR so would love to box in EU PII in EU servers.

This reads like a professor who&#x27;s so steeped in research that he&#x27;s forgotten how to communicate to his students!<p>What exactly are we talking about here? A WebSQL thats actually synced to a proper RDBMS? Synced across devices? I&#x27;m not clear about an end to end use case.<p>Edit: Honestly, this line from the LiteFS docs[0] needs to be added to the top of the article:<p>&gt; LiteFS is a distributed file system that transparently replicates SQLite databases. This lets you run your application like it&#x27;s running against a local on-disk SQLite database but behind the scenes the database is replicated to all the nodes in your cluster. This lets you run your database right next to your application on the edge.<p>I had no idea what was being talked about otherwise.<p>[0]: <a href="https:&#x2F;&#x2F;fly.io&#x2F;docs&#x2F;litefs&#x2F;" rel="nofollow">https:&#x2F;&#x2F;fly.io&#x2F;docs&#x2F;litefs&#x2F;</a>