Congratulations to the automerge team! This is a fantastic accomplishment.<p>I particularly enjoy the performance improvements. I benchmarked automerge 18 months ago and the benchmark took about 5 minutes to process the edits of a paper. Some single character inserts took as much as 2 seconds of cpu time. From the article it looks like this entire editing trace (of 260000 keystrokes) is down to 600ms. That’s a huge improvement. It means automerge is similar in performance to yjs, and in turn that makes automerge useful for a much broader set of applications.<p>One thing I really enjoy about the collaborative editing space is how much ideas are shared around. The highly compact binary encoding was done first in automerge, then copied and tuned in yjs and diamond types. The idea of using an internal list rather than a tree went the other way - yjs came up with the idea, and that approach has landed in all production sequence CRDTs that I know about.<p>There’s a bunch of work in the pipeline around non-interleaving, BFT properties, database interoperability and more performance tuning that we are (collectively) still figuring out. But the future of CRDTs seems bright. In a few years I’d love all new software to be built on local first fundamentals. Work like this is how we get there.<p>To everyone involved, great work! Keep it coming!
See also, Autosurgeon (with a 0.3.0 release today), which is a higher level API on top of Automerge for Rust:<p>I'm building a mobile app with a server backend, and I was looking for resources to build them in an offline-first way (since unlike on the browser, people expect to use apps offline, if they can, such as fitness or habit trackers).<p>I found the concept of conflict-free replicated data types (CRDTS) interesting as it allows you to have fully offline experiences while also having a conflict-free syncing experience. I was looking for some good libraries and came across automerge [0] and yrs [1], but both had some rough APIs as they're primarily low-level Rust libraries that are wrapped by higher-level TypeScript APIs.<p>Autosurgeon wraps the low-level API of automerge to make it much more ergonomic, closer to the TypeScript experience, but in Rust of course. You can for example use `struct`s which autosurgeon will serialize and deserialize automatically, which is not present in base automerge, which focuses more on string keys and arbitrary values.<p>I am planning on using this together with Flutter and flutter_rust_bridge [2] in order to use this same Rust library everywhere. In this case, the server just becomes another (albeit more privileged) client.<p>[0] <a href="https://github.com/automerge/automerge-rs">https://github.com/automerge/automerge-rs</a><p>[1] <a href="https://github.com/y-crdt/y-crdt">https://github.com/y-crdt/y-crdt</a><p>[2] <a href="https://github.com/fzyzcjy/flutter_rust_bridge">https://github.com/fzyzcjy/flutter_rust_bridge</a>
Big congratulations to the Automerge folks for shipping this after years of work. At this point both Yjs and Automerge have Rust libraries and (soon?) bindings for more languages than just JS that stay in sync.<p>Yjs (pure javascript?) is quoted on the paper benchmark at 1,074ms and 10,141,696 bytes of memory, compared to Automerge 2.0.2-unstable at 661ms and 22,953,984 bytes of memory. It looks like Automerge 2 latest is faster than Yjs, but still uses 2x more memory.<p>I wonder if this is comparing usage from JS via bindings, or directly comparing two different rust implementations, or comparing Automerge 2.0.2-unstable via Rust to Yjs via NodeJS.<p>I am still not sure which set of tools I would recommend; I believe Yjs is more actively deployed in production since the Automerge implementation was so far behind performance wise until now. However one of the Peritext authors (<a href="https://twitter.com/sliminality" rel="nofollow">https://twitter.com/sliminality</a> who is on my team at Notion) tells me that Automerge is better at text because it doesn't suffer from interleaved characters like Yjs does. So consider it instead of Yjs!
Related:<p><i>Automerge CRDT – Build local-first software</i> - <a href="https://news.ycombinator.com/item?id=30881016" rel="nofollow">https://news.ycombinator.com/item?id=30881016</a> - April 2022 (8 comments)<p><i>Automerge: A JSON-like data structure (a CRDT) that can be modified concurrently</i> - <a href="https://news.ycombinator.com/item?id=30412550" rel="nofollow">https://news.ycombinator.com/item?id=30412550</a> - Feb 2022 (69 comments)<p><i>Automerge: a new foundation for collaboration software [video]</i> - <a href="https://news.ycombinator.com/item?id=29501465" rel="nofollow">https://news.ycombinator.com/item?id=29501465</a> - Dec 2021 (29 comments)<p><i>Automerge: A library [..] for building collaborative applications in JavaScript</i> - <a href="https://news.ycombinator.com/item?id=24791713" rel="nofollow">https://news.ycombinator.com/item?id=24791713</a> - Oct 2020 (1 comment)<p><i>Automerge: JSON-like data structure for building collaborative apps</i> - <a href="https://news.ycombinator.com/item?id=16309533" rel="nofollow">https://news.ycombinator.com/item?id=16309533</a> - Feb 2018 (98 comments)
So exciting! Strangely enough, a couple of hours before this release, we just managed to wrap our heads around Yjs after playing with it on and off for a few weeks!<p>For anyone not up to date with the world of CRDTs, Seph Gentle's two blog posts have become legendary:<p>* <a href="https://josephg.com/blog/crdts-are-the-future/" rel="nofollow">https://josephg.com/blog/crdts-are-the-future/</a><p>* <a href="https://josephg.com/blog/crdts-go-brrr/" rel="nofollow">https://josephg.com/blog/crdts-go-brrr/</a><p>these are also worth checking out:<p>* <a href="https://github.com/y-crdt/y-crdt">https://github.com/y-crdt/y-crdt</a> (rust implementation started by the creator of Yjs, Kevin Jahns)<p>* <a href="https://github.com/y-crdt/ypy">https://github.com/y-crdt/ypy</a> (python bindings for the rust implementation)<p>* <a href="https://github.com/josephg/diamond-types">https://github.com/josephg/diamond-types</a> (Seph Gentle's rust implementation of YATA, the algorith behind Yjs)
As it stands CRDTs are only really useful for a narrow subset of data. Data is guaranteed to converge, but there is no guarantee that the final result makes any kind of semantic sense in the application domain.<p>One can write custom conflict resolution and treat the data structures as a convenient baseline for event sourcing, but that requires a lot of work and potentially often user guided resolution.<p>I'd really love to see some research into deriving CRDT merge semantics from a formal description of application behaviour.
Sorry if this is a stupid question but how do I write the json changes to a MySQL database via Php? From what I can understand the json is being updated on the client side via javascript.. but to save the changes do I have to send the entire json doc to server or is it possible to somehow patch the json on php side also? i want to create a google docs type autosave functionality so sending the whole json seems quite wasteful.<p>I'm learning web programming and this seems quite useful for what I'm doing. Any tips on how to do it in php with this?
I am currently using yjs. What would be the equivalent way as described here [0] for yjs to sync docs in Automerge? I don't need any WebSockets or real time stuff. It always seemed so complicated in Automerge compared to yjs. I just want to roll my own simple sync mechanism via HTTP.<p>[0]: <a href="https://docs.yjs.dev/api/document-updates#syncing-clients" rel="nofollow">https://docs.yjs.dev/api/document-updates#syncing-clients</a>
Great improvements, performance was probably the major concern of automerge prev versions, I have begin reading about CRDTs and this is the correct movement field at the moment.<p>Local first in my mind means your users can still work even if your server is down for a small amount of time for any reason, which means the users can also jump on an airplane and decide to review/update their price table or check out their sales commissions if they want.
It benefits both sides of the software economy (users and providers)[1].<p>I recommend the reading the paper from 2019 about local first it is not too academic and gives a good view of the challenges [2].<p>[1] <a href="https://mateusfreira.github.io/@mateusfreira-2022-12-04-my-thoughts-on-local-first-software/" rel="nofollow">https://mateusfreira.github.io/@mateusfreira-2022-12-04-my-t...</a>
[2] <a href="https://martin.kleppmann.com/papers/local-first.pdf" rel="nofollow">https://martin.kleppmann.com/papers/local-first.pdf</a>
Excited to see this!<p>I built a personal project with automerge 1 recently because I liked the philosophy of offline-first from the team and because the docs were frankly much more approachable than yjs but ended up switching to yjs half way through because of performance issues and also for the rich text support (via the delta doc type).<p>A little bummed about 2.0 not working with react native because of webassembly but excited to see the peritext work progress for rich text coming soon.<p>CRDT is one of those technologies I assumed was “done” a decade ago and I was surprised to learn how much of the major progress was only just recently made when I dug in.<p>Definitely expecting to see some cool new multiplayer startups built on this tech.
Would be great to have a CDN-based JS distribution for those who want to play without the heartache of JS build systems (npm/yarn/webpack/etc).
For now we use pouch/couch for this purpose, which does this (merge docs automatically and pick a winner) out of the box, but has the disadvantage of having to run couch which is an infrastructure pain. We have been exploring substituting it with crdt and this release seems to be the sign of maturity we needed to get us over the line.
> Cloud software is fragile and prone to outages, rarely supports offline use, and is expensive to scale to large audiences.<p>Hint to whoever wrote this: except the offline thing, this has not been my experience at all. Automerge sounds cool enough as is, no need to make up reasons why Cloud Bad.
Both y-crdt and Automerge are rust libraries (though the latter seems to have a lot more targets), is there a short rundown/vs comparison somewhere? This article compares performance with yjs, but not y-crdt, and there are probably other comparison points to be studied.
This is really cool. Theres a large class of applications for which deterministic conflict resolution makes a lot of sense. Having a mature library available and not having to implement your own versions from research papers is great.<p>At least in theory.. anyone used this?
This is awesome, especially the massive improvements to performance: The fact that a CRDT file can be within 2x the size of a plain text file whilst still fully loading within 2s is a great sight to behold.
Can you cram a duck DB in this? Or maybe SQLite?
I suppose you can export several tables into jsonlines files, but it might be cool to have a cleaner solution on the backend.
Is advised to use CRDTs for an offline app that syncs to a central server, such as a todo app like Apple's Reminders or Todoist? Can simpler methods suffice ?
I loved Automergr and they chose all the right tech. But they disclosed a huge caveat: the system got massively slower as more operations were done on the CRDT. Is that still the case or was it fixed in 2.0?
Link isn't working (extra /), <a href="https://automerge.org/blog/automerge-2/" rel="nofollow">https://automerge.org/blog/automerge-2/</a>
> you can just think of it as a version controlled data structure. Automerge lets you record changes made to data and then replay them in other places<p>This is not my understanding of what a CRDT is. From Wikipedia:<p>> [CRDTs] send their full local state to other replicas, where the states are merged by a function which must be commutative, associative, and idempotent. The merge function provides a join for any pair of replica states, so the set of all states forms a semilattice. The update function must monotonically increase the internal state, according to the same partial order rules as the semilattice.<p>For those people saying there is no such thing as conflict-free, there is! But only for datatypes that satisfy these constraints.
Any clue when the team will be releasing the swift version?
I've been experimenting with the 1.x release to use in a Markdown editor but I had to revert for now to using my own CRDT implementation. Mainly because of speed (caching is hard but useful) and API limits (e.g. need to update cursor, and index text in documents). Still I love the abstraction of Automerge that I can treat as a black box container that I can just trust to do the right thing in the most optimal manner. It's a very nice abstraction.<p>ref to my implementation:
<a href="https://github.com/munhitsu/CRAttributes">https://github.com/munhitsu/CRAttributes</a>
<a href="https://github.com/munhitsu/CRAttributesDemo">https://github.com/munhitsu/CRAttributesDemo</a>
Can someone answer how we go about the single-source-of truth problem in this distributed scenario? Or does this approach guarantee consistency among all synced data so that they are all effectively the same?<p>I can see how this would be used for 'live data sharing' but what about for more persistent information, like documents, designs etc?