TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

Simple declarative schema migration for SQLite

109 点作者 drothlis大约 3 年前

9 条评论

simonw大约 3 年前
For anyone who needs to implement the 12 step procedure (<a href="https:&#x2F;&#x2F;www.sqlite.org&#x2F;lang_altertable.html#otheralter" rel="nofollow">https:&#x2F;&#x2F;www.sqlite.org&#x2F;lang_altertable.html#otheralter</a>) from the SQLite documentation for applying complex alters, I&#x27;ve built a CLI tool and Python library that can apply that for you.<p>CLI example: <a href="https:&#x2F;&#x2F;sqlite-utils.datasette.io&#x2F;en&#x2F;stable&#x2F;cli.html#cli-transform-table" rel="nofollow">https:&#x2F;&#x2F;sqlite-utils.datasette.io&#x2F;en&#x2F;stable&#x2F;cli.html#cli-tra...</a><p><pre><code> sqlite-utils transform fixtures.db roadside_attractions \ --rename pk id \ --default name Untitled \ --type longitude float \ --type latitude float \ --drop address </code></pre> Python example: <a href="https:&#x2F;&#x2F;sqlite-utils.datasette.io&#x2F;en&#x2F;stable&#x2F;python-api.html#python-api-transform" rel="nofollow">https:&#x2F;&#x2F;sqlite-utils.datasette.io&#x2F;en&#x2F;stable&#x2F;python-api.html#...</a><p><pre><code> table.tranform( rename={&quot;pk&quot;:&quot;id&quot;}, defaults={&quot;name&quot;: &quot;Untitled&quot;}, types={ &quot;latitude&quot;: float, &quot;longitude&quot;: float }, drop={&quot;address&quot;} ) </code></pre> Wrote more about those here: <a href="https:&#x2F;&#x2F;simonwillison.net&#x2F;2020&#x2F;Sep&#x2F;23&#x2F;sqlite-advanced-alter-table&#x2F;" rel="nofollow">https:&#x2F;&#x2F;simonwillison.net&#x2F;2020&#x2F;Sep&#x2F;23&#x2F;sqlite-advanced-alter-...</a><p>(Just noticed this is already mentioned at the bottom of the blog entry!)
评论 #31251838 未加载
simonw大约 3 年前
This is an interesting approach to this problem.<p>I like the idea of building the new schema as an in-memory database. You could even go a step further and compare them using a SQL query that joins across the two databases (SQLite supports this, it&#x27;s pretty neat).<p>I&#x27;m a bit nervous about how edge-cases might screw things up, but a nice thing about SQLite is that backups are really cheap so I guess you can protect against any risks by creating a backup before running this script.<p>My preferred approach to database migrations is the one implemented by Django: migrations are a sequence of transformations stored in files on disk, and the database includes a table that says which of those migrations have been applied. This keeps everything in version control and means there&#x27;s no chance of the wrong migration being applied in the wrong way.<p>It&#x27;s quite a bit more work to setup though. I don&#x27;t have my own Django-style migration system for SQLite yet and I really need one.<p>I really love how clean and short the implementation of this is! <a href="https:&#x2F;&#x2F;david.rothlis.net&#x2F;declarative-schema-migration-for-sqlite&#x2F;migrator.py" rel="nofollow">https:&#x2F;&#x2F;david.rothlis.net&#x2F;declarative-schema-migration-for-s...</a>
评论 #31251857 未加载
评论 #31252103 未加载
评论 #31253162 未加载
wmanley大约 3 年前
Disclosure: I&#x27;m the co-author of the blog post.<p>One nice thing about this approach is that because it works by inspecting a database that has been put in a desired state. This means it doesn&#x27;t matter how you set up your database (with an ORM, a SQL file, etc.) the same function can be used to do the migration.<p>I&#x27;d really like if something like this were built into SQLite. That way we&#x27;d have more confidence that it would be&#x2F;stay correct and complete. It seems generic enough that it could be a part of built-in functionality. I would have proposed it upstream, but AIUI the SQLite authors don&#x27;t accept external patches.<p>Edit: Another benefit of this being built-in: there are plenty of places where SQLite is applicable where you can&#x27;t, or don&#x27;t want to run Python.
评论 #31254745 未加载
评论 #31255680 未加载
harryvederci大约 3 年前
Interesting approach, I&#x27;m taking notes!<p>Something I&#x27;m currently doing for my CV application[0] is auto-generating variables for every table&#x2F;column name in my SQLite DB. As I&#x27;m using a dynamically typed language, it&#x27;s normally easy to mess up with a typo when getting a valuable out of a query result. But when I use the auto-generated variables I&#x27;ll know I messed up during compile time. That also means my SQLite schema is always my source of truth. Maybe there are better ways to do it, but I find it really useful so far!<p>[0] <a href="https:&#x2F;&#x2F;withoutdistractions.com&#x2F;cv" rel="nofollow">https:&#x2F;&#x2F;withoutdistractions.com&#x2F;cv</a> - I&#x27;m doing a &quot;Show HN&quot; today here: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=31246696" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=31246696</a><p>(I have to admit I&#x27;m kinda jaleous you beat me to the front page within half an hour. What sacrifice did you make to which algorithm deity? I won&#x27;t tell anyone.)
评论 #31250834 未加载
Sytten大约 3 年前
Nice tool! One common problem I have with SQLite is the DDL to alter tables is quite limited and the performance is not super. I actually opened <a href="https:&#x2F;&#x2F;sqlite.org&#x2F;forum&#x2F;forumpost&#x2F;8b4e95fd55" rel="nofollow">https:&#x2F;&#x2F;sqlite.org&#x2F;forum&#x2F;forumpost&#x2F;8b4e95fd55</a> a few days ago because it is currently a pain to drop a column that has a foreign key constraint.<p>Concerning performances, I am dealing with sqlite databases of a couple GB (5-20) and dropping a column can easily take 15-20m plus it doubles the size of the table on disk. If someone has tips on how to improve that I am interested!
评论 #31255175 未加载
billllll大约 3 年前
We had something similar at Google, where you change some CREATE TABLE statements in an SQL file, then run a migration based on that changed SQL file. It was insanely nice. I think this tool can be super useful, especially with a lot more people using something like Litestream.<p>I&#x27;m personally (and maybe naively) of the opinion that DB migrations should be as painless&#x2F;automated and without ceremony as possible. I wish there were more tools like this out in the wild.
评论 #31251967 未加载
jitl大约 3 年前
I think this is an interesting way to generate migration scripts at design time, but I wouldn&#x27;t trust auto-migration code in production that migrates from arbitrary schema to arbitrary schema. Take migrations that require copying the entire table - this could be fine for small tables, but extremely expensive for large tables. If disk space is a concern, this will also use 2x disk space! I at least want any such operations to be <i>obviously visible</i> at code review time, so the team can discuss any performance implications before we ship such a migration.<p>Our team uses a rigid workflow to maintain the SQLite schemas for our mobile and desktop apps:<p>1. Generate a new migration. This creates ${SCHEMA_VERSION}-migration-name.sql, where SCHEMA_VESION will be the PRAGMA user_version after migration. We also generate a matching ${SCHEMA_VERSION}-migration-name.test.ts.<p>2. The developer updates the migration file to do the thing. This part we could replace with declarative table diffing as in the article, which would be cool! The developer also fills in the test. Our tests typically write some rows to the DB at SCHEMA_VERSION-1, perform the migration, and then make assertions about how the rows look after the migration.<p>3. When the changes are ready for review, the developer runs a command that packs up all the migrations into generated allMigrations.{json,ts} files, along with an auto-generated &quot;fast-forward&quot; migration that is just the final database SQL dump, and a JSON description of the expected end result schema.<p>4. On end-user devices, at app boot we read PRAGMA user_version and apply the migrations transactionally in order, to the latest migration. At the end, we assert that the schema in the database matches the expected schema. If the database is new, like on a fresh install, we apply the fast-forward migration instead.
bvrmn大约 3 年前
Related project: Alembic[1]. It has tight integration with SQLAlchemy and allows autogenerate migrations based on code&#x2F;db changes.<p>[1]: <a href="https:&#x2F;&#x2F;alembic.sqlalchemy.org&#x2F;en&#x2F;latest&#x2F;" rel="nofollow">https:&#x2F;&#x2F;alembic.sqlalchemy.org&#x2F;en&#x2F;latest&#x2F;</a>
tenken大约 3 年前
Why are all these tools written in python? I&#x27;m not complaining, sometimes I&#x27;d like to see such a tool in Php or Bash or something where say my project doesn&#x27;t need to include yet another Language dependency ... Doesn&#x27;t SQLite cli for example ship as a bash or win32 CMD file...?