I think I saw an Uber-engineer's presentation (YouTube; can't find it now anymore) about their test environment having the ability to automatically fork the database for every new test run, such that they could easily "revert" (in their case ignore/walk away from) the database after a test run. I find that an appealing idea, since regular test runs in the places where I work, tend to create cruft data in the test enviroment that is lopsided and voluminous in ways that do not properly represent the production system anymore. (And no one wants to troubleshoot performance issues that only ever exist in the test environment.) I have to imagine that pg_branch might be part of building a solution similar to this? Could people speak to some practical applications for this? Maybe I'm thinking about entirely the wrong one here...
For anyone wondering what "Neon-like branching" is, this is worth a read: <a href="https://neon.tech/blog/database-branching-for-postgres-with-neon" rel="nofollow noreferrer">https://neon.tech/blog/database-branching-for-postgres-with-...</a>
How does this differ with Postgres AI's thin cloning and database branching?
<a href="https://github.com/postgres-ai/database-lab-engine">https://github.com/postgres-ai/database-lab-engine</a>
Interesting that branching is now better supported and almost free. I wonder if merging can be simplified or whether it already is as simple and as fast as it can be?<p>I guess I am inspired by Dolt’s ability to branch and merge: <a href="https://github.com/dolthub/dolt">https://github.com/dolthub/dolt</a>
Holy hell this thing is awesome. I am incredibly surprised at how little code it took to achieve a MVP on this. Adding this to local docker image would make doing so many things easier during testing and local dev.<p>Having it on CI actions would also be amazing.
Don't use BTRFS! I <i>still</i> encounter crippling data loss and error messages in my dmesg log after giving it another shot for the last 10 months. I try btrfs with Arch Linux every few years because of its native transparent compression abilities. Enough is enough! Just say no. Friends don't let friends use btrfs.<p>bcachefs is the only hope left for normal Linux users but it has not landed yet. I am using ZFS for /home (with zstd compression) on LVM on LUKS and my root partition is ext4 on LVM on LUKS.<p>I considered VDO for LVM because it supports compression but VDO IS A JOKE! <a href="https://github.com/dm-vdo/vdo/issues/38">https://github.com/dm-vdo/vdo/issues/38</a>
If interested in using Neon for ephemeral environments, Coherence (I’m a cofounder) recently launched a first-class integration with Neon for just this purpose: <a href="https://docs.withcoherence.com/docs/reference/integrations#neon" rel="nofollow noreferrer">https://docs.withcoherence.com/docs/reference/integrations#n...</a><p>Would be cool to try using this extension on top of RDS or Cloud SQL to offer the same experience without the overhead of another SaaS tool!
Author here: thanks for taking a look at pg_branch! I didn't expect to have this posted to HN today, but I'm happy to answer any questions folks have about it.
One thing I could see where this could be useful would be unit test suites where you want different sets of seed data, but you want to be sure that each test doesn't accidentally mutate and leak state to other tests. There are other ways to do this, but this seems like it might be a natural fit for isolating persistence changes.
For ad hoc copies I’ve found the third party schema cloning function useful. I assume performance would be an issue for huge databases though. <a href="https://github.com/denishpatel/pg-clone-schema">https://github.com/denishpatel/pg-clone-schema</a>
I once saw another extension, or product on top of postgres, that achieved copy-on-write semantics (and I think it didn't require a fs with snapshots like btrfs). It wasn't pg_branch or pgcow. Does anyone heard of something like this?
Really cool, I've implemented something similar for MySQL on top of Ubuntu and LVM snapshots. With a very low space overhead, we were able to create tens of copies of the main database for development purpose.
This is neat, but I wouldn't trust BTRFS for any serious usage. I've read way too many data loss reports to make me stay away even for personal use.<p>Could this work on ZFS, or a layered FS like Aufs/OverlayFS?
Remind me why macOS is the best development OS again...? This has the potential to 2x my debug speed when bisecting test failures and testing fixes. Wonder what corporate IT will say if they see me doing real work on a Linux vm, I'm <i>very</i> tempted to check.<p>Edit: quick google tells me APFS supports snapshots/cow, could the same approach be used here?