Some notes on local-first development

As someone who has been doing local-first for the last 3 years with Notesnook, let me tell you: it&#x27;s not all gardens and roses. Local first has its own very unique set of problems:<p>1. What to do with stale user data? What happens if a user doesn&#x27;t open the app for a year? How do you handle migrations?<p>2. What about data corruption? What happens if the user has a network interruption during a sync? How do you handle partial states?<p>3. What happens when you have merge conflicts during a sync? CRDT structures are not even close to enough for this.<p>4. What happens when the user has millions of items? How do you handle sync and storage for that? How do you handle backups? How do you handle exports?<p>One would imagine that having all your data locally would make things fast and easy, but oh boy! Not everyone has a high end machine. Mobiles are really bad with memory. iOS and Android have insane level of restrictions on how much memory an app can consume, and for good reason because most consumer mobile phones have 4-6 gbs of RAM.<p>All these problems do not exist in a non-local-first situation (but other problems do). Things are actually simpler in a server-first environment because all the heavy lifting is done by you instead of the user.

I was exploring an interesting local-first component earlier today: cr-sqlite, which adds CRDTs to SQLite so you can have multiple databases that receive writes and then sync up and merge their changes later on.<p>My notes here: <a href="https:&#x2F;&#x2F;til.simonwillison.net&#x2F;sqlite&#x2F;cr-sqlite-macos" rel="nofollow noreferrer">https:&#x2F;&#x2F;til.simonwillison.net&#x2F;sqlite&#x2F;cr-sqlite-macos</a>

What even is “local-first”.<p>I’m a big proponent of being able to run the code locally, but this seems to be more along the lines of, first update some local DB, then sync it.<p>I see nothing fundamentally different here than using a well designed API for ‘sync’, and none of it does anything to revive the web browsing experience. Gone are the days where “page load” meant that you could safely begin reading without interruption. This whole “web application” thing was a foobar’d mess. Yes, we probably want some level of interactivity, but I fail to see how casting the web application problem as a database sync issue addresses the fundamental issues.<p>That being said, I can see the appeal. It’s a clean interface to work with and it allows for new frameworks to displace existing problematic frameworks.<p>I just worry that in 5 years we’ll all be complaining about poorly designed schema and overloaded table updates again. The solution? Microservices with their own smaller tables which batch updates to the larger system of course!

The post writes about multiplayer, especially at the end. In games, but also in basically every other application, there is a state known to players (the cards in hand, the map around, etc) and a state that must be secret (the cards of the other players, the rest of the map, etc.) The server with the knowledge of all the state can be a client like the others but it has access to the real database. It can have the code to resolve interactions between players or that code runs locally to players too.<p>In complex turn based games there is a lot to do locally before sending the move to the server. Think about moving tens of units on a map. The problem here is of there are unknown parts of the map to explore. You need a connection to the server. However anything else will run faster because it doesn&#x27;t have to hit the network and a round trip to the database on the server.<p>The post writes about Figma being developed local first. You don&#x27;t have to hit the network while drawing, if you don&#x27;t have to collaborate with someone on a shared document. Then you send the state to the server and your private state becomes known to all the other coworkers.

This is a great article. I&#x27;ve been working on tooling related to local-first&#x2F;offline-first for the last 10 years, and the benefits both for end-users and developers are great.<p>The biggest benefit for developers that I&#x27;ve seen is that state management gets _much_ simpler when it is all handled by a local database. Just run queries directly on every app&#x2F;page, with live queries if the framework supports it. No need for maintaining in-memory app state and propagating changes to the different parts, no need for caching and associated cache invalidation issues.<p>The caveat is that keeping the local database in sync with the server is not easy, and it&#x27;s best to use other tooling for it. I&#x27;m working on PowerSync [1] myself, and ElectricSQL [2] also looks quite promising (both also mentioned in OP).<p>[1]: <a href="https:&#x2F;&#x2F;powersync.co" rel="nofollow noreferrer">https:&#x2F;&#x2F;powersync.co</a><p>[2]: <a href="https:&#x2F;&#x2F;electric-sql.com" rel="nofollow noreferrer">https:&#x2F;&#x2F;electric-sql.com</a>

About 12+ years ago, i &quot;invented&quot; similar RPC-over-couchDB architecture as there wasn&#x27;t much other distributed ways.<p>Worked wonders - each user reading&#x2F;posting requests&#x2F;responses in own &quot;channel&quot;, the channel being replicated to server and any of user&#x27;s devices, central server responding to requests and eventually cross-polinating the channels if need be.<p>Did a web client (pouchDB), android (touchDB), ios (forgot the name).. essentially each platform has its own implementation of the relatively simple protocol of Couchdb. So all one does is listen&#x2F;read&#x2F;write to a single local db.. and then db is doing all the communication, whenever possible. Simplified the clients sooooo much - reduced work and complexity like 50%.<p>Eh, the thing didn&#x27;t go anywhere as product but nevermind :&#x2F;<p>have fun

A whole new generation of developers is learning what the previous generation termed &quot;briefcase applications&quot;. When client-server applications were the thing late 90s and early 2Ks, internet speeds were a serious limitation. This <i>forced</i> many architectures to work with &quot;local-first&quot;, disconnected dataset, eventually-synchronised desktop applications. Microsoft ADO famously touted the &quot;disconnected dataset&quot; for this after Borland&#x27;s &quot;client dataset&quot; pioneered this concept for the Windows desktop. Eventually even Java got disconnected datasets. All these techs more than two decades ago had real practical problems they solved: one of them I worked on involved collecting water flow data from several thousands of rivers in Southern Africa. Hundreds of users had mobile laptops that fired up desktop apps with authentication and business rules. Users then came back to head office to synchronise or used a branch with network to upload.<p>They worked and they were necessary.<p>Things changed when internet connectivity eventually became cheap and ubiquitous and the complexity of developing such applications then didn&#x27;t merit the effort.<p>Now, the swing back to &quot;local-first&quot; is mainly for user-experience reasons. The same theoretical complexities of &quot;eventually synchronised&quot; that existed for the &quot;briefcase app&quot; are still present.<p>Is the complexity worth it though?

This is about offline-first applications. I first thought it was about developing on localhost instead of in the cloud...

There are use cases that can benefit from this, and I built a saas door access control system based around some of these ideas over 10 years ago.<p>However, the vast majority of apps don’t need this. Please don’t do anything like this for your boring enterprise CRUD app (60 fps CRUD? What even?).<p>I know many of you will, so I guess I’ll just console myself with the likelihood I’ll have a ton of consulting work in the next 5-10 years.

There&#x27;s an older term for &quot;local-first&quot;, aka, <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Rich_client" rel="nofollow noreferrer">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Rich_client</a><p>The thin&#x2F;thick client cycle starts again...

I work for Ably, a realtime messaging platform. I&#x27;m also the author of SQLEdge (a replication tool mentioned in the article). One of our teams at Ably just launched &quot;spaces&quot;[1] to make it easy to build collaborative experiences. While it&#x27;s not strictly in the local-first camp, I think it&#x27;s in a similar &#x27;space&#x27; (pun intended).<p>I think the collaboration experiences that you see in Figma, Miro, Notion, etc should _just be possible_ for all apps. And for a lot of product development teams they probably look at what would be involved to try and build a realtime collaboration experience and think it&#x27;s just gonna be either too hard, or too much work. But the &#x27;art of the possible&#x27; as definitely moved on (as evidenced by the list of companies in the blog post that are addressing these challenges).<p>As a bunch of the other commenters point out, the hard-part isn&#x27;t just building the code for the UI that will show collaboration experiences, the hard-part is often handling the stateful connections (websockets) and making sure the messages are delivered to the right clients. You either build a stateless&#x2F;polling mechanism over HTTP, or you have to deal with trying to scale and load-balance websocket connections. Neither approach is particularly easy.<p>[1] <a href="https:&#x2F;&#x2F;ably.com&#x2F;spaces" rel="nofollow noreferrer">https:&#x2F;&#x2F;ably.com&#x2F;spaces</a>

Does someone has a successful local-first product used by paying customers?<p>I am interested in your take on local-first.

I like the idea but I&#x27;ve not tried it, it just seems quite a bit more complicated than just accepting the need to be online.<p>I&#x27;m talking hobby projects here where my main problem is getting it to a usable state so that it works (for me) without quirks and actually solves my problem instead of being a proof of concept.<p>Assuming the data model is kinda boring I wonder if there&#x27;s a way to leverage some &quot;framework&quot; to offload the syncing part, for a simple CRUD app. There should be no resolution strategies, it has like a users table, a posts table and a comments table (classic blog example), this should be easily abstractable, or not? But then most people are talking about Websites as frontends, I guess, which changes my current &quot;I&#x27;m gonna write a Qt native client&quot; idea.

Hearing the name, I first thought this was about reducing complexity and removing unnecessary dependencies on cloud services. But it seems rather about adding yet another layer of complexity: You still have to track the state on the server, but now you want to <i>also</i> track it on the client.

Tried to get pouchdb &#x2F; rxdb working on a number of projects over the years. Hoodie also. Ran into &quot;too hard&quot; problems every time and fell back to something else :&#x2F;<p>I think what I liked the least was the complexity introduced to the client (a fun statement considering some of the apps I&#x27;ve worked on recently). Instead of being able to abstract away business logic behind a simple API, it was there in all it&#x27;s &quot;haute logique&quot; glory - hardwired into my otherwise dead simple todo app.<p>That said, still love the concept. Keen to try some of the new names. Tho the idea of it living in a worker, where it can be queried &#x2F; controlled &#x2F; deployed (or replaced) as if remote, has a nice ring to it?

The JavaScript&#x2F;webapp world will soon have reached its goal and have a complete operating system running the browser. Then someone will come along and develop some kind of browser in that operating system and the whole game starts again …

This is an awesome resource! Glad Triplit made the list of hosted services (we&#x27;re also open-source)<p>One common misconception with local-first, is that it&#x27;s essentially local-only. But I believe the best experience is where the server and client are equally capable and only a partial set of data is stored on the client. The server still maintains authority but the client is always snappy when updated data or changing queries&#x2F;views. We basically call this a full-stack database and we&#x27;re building that at [Triplit](www.triplit.dev).

wow, I never heard of &quot;60 FPS CRUD&quot;... that must be next level of form filling efficiency :-p

The analogy is Traditional backend development is thin client software, reactive frameworks are networked software, and local first is the native software (keep data with the client). The browser is the OS.<p>Why not skip the whole App Store generation that’s to come and build a browser extension today?

If anyone will be in St Louis for Strange Loop next week come out to the LoFi Unconf to talk more about local first.<p><a href="https:&#x2F;&#x2F;lu.ma&#x2F;localfirstswunconf-stlouis" rel="nofollow noreferrer">https:&#x2F;&#x2F;lu.ma&#x2F;localfirstswunconf-stlouis</a>

This is what they call a &quot;desktop application&quot;

This is sometimes called &quot;LoFi&quot; development these days and there&#x27;s a Discord where people discuss it:<p><a href="https:&#x2F;&#x2F;localfirstweb.dev&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;localfirstweb.dev&#x2F;</a><p>Despite the name it&#x27;s not really a web exclusive community.<p>Actually the web isn&#x27;t a great way to do this, there&#x27;s alternate stacks that maybe can work better. The Android stack has been rewritten in recent years by Google and parts are now usable on desktop and iOS. Someone posted this to the Kotlin slack earlier today:<p><a href="https:&#x2F;&#x2F;medium.com&#x2F;@mike_21858&#x2F;auto-generating-an-http-server-and-offline-first-data-layer-based-on-a-room-database-using-kotlin-a24f376db7ef" rel="nofollow noreferrer">https:&#x2F;&#x2F;medium.com&#x2F;@mike_21858&#x2F;auto-generating-an-http-serve...</a><p><i>I’ve been working on apps for limited connectivity environments such as Afghanistan and rural Rwanda for many years. I think we have to make offline-first a lot easier than it is now. Our own app Ustad Mobile has more than 60 tables in the database. Writing manual logic to run all of those offline would 1) exceed the resources we have and 2) be prone to human error.<p>This article introduces Door: our open-source way to automate the generation of an offline-first data layer using Kotlin Multiplatform and Kotlin Symbol Processing (KSP). Door is still a work-in-progress and not ready for production use in other apps yet, but we think it’s a game-changer for offline-first development. Feedback on the concept and API is welcome. Door can automatically generate HTTP server endpoints and an offline first client based on a Room database.</i><p>The idea is that you can use Kotlin for both server, Android and also a desktop client, and with KMP you&#x27;re also able to use it on iOS for the backend (UI is still in Swift). So you can share your domain model across all devices and the underlying sync protocols and logic can be auto generated using a compiler plugin. The underlying DB is SQLite wrapped using a library from Google called Room which makes SQLite a bit easier to use.<p>Deploying such an app on Android is obvious, for iOS you need to learn about Kotlin&#x2F;Multiplatform but obviously SQLite can run on iOS just fine and Kotlin can interop with Objective-C&#x2F;Swift. For desktop, deployment is easy if you use Conveyor, which the Ustad guys are planning to do, as it supports Kotlin desktop apps out of the box. You do need to buy signing certificates (or distribute via the app stores) but that&#x27;s a one time cost.<p>The main issue here is if you&#x27;re wedded to the web. Probably stuff can be done with WASM but it&#x27;s not going to be as natural as just using the tech on the JVM.