TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

Show HN: Pg_jsonschema – A Postgres extension for JSON validation

203 点作者 oliverrice将近 3 年前
pg_jsonschema is a solution we&#x27;re exploring to allow enforcing more structure on json and jsonb typed postgres columns.<p>We initially wrote the extension as an excuse to play with pgx, the rust framework for writing postgres extensions. That let us lean on existing rust libs for validation (jsonschema), so the extension&#x27;s implementation is only 10 lines of code :)<p><a href="https:&#x2F;&#x2F;github.com&#x2F;supabase&#x2F;pg_jsonschema&#x2F;blob&#x2F;fb7ab09bf6050130e8d656f2999ec0f6a3fedc0d&#x2F;src&#x2F;lib.rs#L1-L13" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;supabase&#x2F;pg_jsonschema&#x2F;blob&#x2F;fb7ab09bf6050...</a><p>happy to answer any questions!

17 条评论

hardwaresofton将近 3 年前
This is awesome -- really excited that Supabase is picking this up with their commitment to open source in general and PG in particular.<p>Some prior art:<p>- <a href="https:&#x2F;&#x2F;github.com&#x2F;gavinwahl&#x2F;postgres-json-schema" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;gavinwahl&#x2F;postgres-json-schema</a> (mentioned in the repo)<p>- <a href="https:&#x2F;&#x2F;github.com&#x2F;furstenheim&#x2F;is_jsonb_valid" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;furstenheim&#x2F;is_jsonb_valid</a><p>pgx[0] is going to be pretty revolutionary for the postgres ecosystem I think -- there is so much functionality that would benefit from happening inside the database and I can&#x27;t think of a language I want to use at the DB level more than Rust.<p>[0]: <a href="https:&#x2F;&#x2F;github.com&#x2F;tcdi&#x2F;pgx" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;tcdi&#x2F;pgx</a>
评论 #32185563 未加载
dmitry_dygalo将近 3 年前
The `jsonschema` crate author here.<p>First of all, this is an exciting use case, I didn&#x27;t even anticipate it when started `jsonschema` (it was my excuse to play with Rust). I am extremely pleased to see such a Postgres extension :)<p>At the moment it supports Drafts 4, 6, and 7 + partially supports Draft 2019-09 and 2020-12. It would be really cool if we can collaborate on finishing support for these partially supported drafts! What do you think?<p>If you&#x27;ll have any bug reports on the validation part, feel free to report them to our issue tracker - <a href="https:&#x2F;&#x2F;github.com&#x2F;Stranger6667&#x2F;jsonschema-rs&#x2F;issues" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;Stranger6667&#x2F;jsonschema-rs&#x2F;issues</a>.<p>Re: performance - there are a couple of tricks I&#x27;ve been working on, so if anybody is interested in speeding this up, feel free to join here - <a href="https:&#x2F;&#x2F;github.com&#x2F;Stranger6667&#x2F;jsonschema-rs&#x2F;pull&#x2F;373" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;Stranger6667&#x2F;jsonschema-rs&#x2F;pull&#x2F;373</a><p>P.S. As for the &quot;Prior Art&quot; section, I think that <a href="https:&#x2F;&#x2F;github.com&#x2F;jefbarn&#x2F;pgx_json_schema" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;jefbarn&#x2F;pgx_json_schema</a> should be mentioned there, as it is also based on `pgx` and `jsonschema`.
评论 #32190695 未加载
rubenfiszel将近 3 年前
This is absolutely brilliant.<p>In windmill, <a href="https:&#x2F;&#x2F;github.com&#x2F;windmill-labs&#x2F;windmill" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;windmill-labs&#x2F;windmill</a> (self-hostable AWS Lambda, OSS AGPLv3) we infer the jsonschema of your script by doing static analysis but so far we were not doing validation of the payload itself, if your script failed because of incorrect payload that was your problem. Now without any additional effort I will be able to add validation and great error reporting &quot;for free&quot;.
评论 #32183642 未加载
paulmd将近 3 年前
Nice work, however, I am structurally dubious of putting too much functionality onto a classical centralized RDBMS since it can&#x27;t be scaled out if performance becomes a problem. It&#x27;s CPU load and it&#x27;s tying up a connection which is a large memory load (as implemented in postgres, connections are &quot;expensive&quot;) and since this occurs inside a transaction it&#x27;s holding locks&#x2F;etc as well. I know it&#x27;s all compiled native code so it&#x27;s about as fast as it can be, but, it&#x27;s just a question of whether it&#x27;s the right place to do that as a general concern.<p>I&#x27;d strongly prefer to have the application layer do generic json-schema validation since you can spawn arbitrary containers to spread the load. Obviously some things are unavoidable if you want to maintain foreign-key constraints or db-level check constraints&#x2F;etc but people frown on check constraints sometimes as well. Semantic validity should be checked before it gets to the DB.<p>I was exploring a project with JSON generation views inside the database for coupling the DB directly to SOLR for direct data import, and while it worked fine (and performed fine with toy problems) that was just always my concern... even there where it&#x27;s not holding write locks&#x2F;etc, how much harder are you hitting the DB for stuff that, ultimately, can. be done slower but more scalably in an application container?<p>YAGNI, I know, cross the bridge when it comes, butjust as a blanket architectural concern that&#x27;s not really where it belongs imo.<p>In my case at least, probably it&#x27;s something that could be pushed off to followers in a leader-follower cluster as a kind of read replica, but I dunno if that&#x27;s how it&#x27;s implemented or not. &quot;Read replicas&quot; are something that are a lot more fleshed out in Citus, Enterprise, and the other commercial offerings built on raw Postgres iirc.
评论 #32182227 未加载
评论 #32181970 未加载
评论 #32192770 未加载
评论 #32195238 未加载
banashark将近 3 年前
Very cool!<p>I remember when kicking the tires on postgrest&#x2F;postgraphile that I found validation and error handling to be one of the less intuitive areas. Not the actual field-level constraints, but how to adapt it to fit a fast-fail vs slow-fail model.<p>When I had attempted before, the only ergonomic option was fast-fail (the first check constraint violated would bubble the error upward) rather than slow-fail (collect all invalid fields and return the collection of errors, which IME is more common on average web forms or api requests).<p>Looking at the single code file and tests, I see only singular field errors. Has a more ergonomic approach to validation-error collection been developed other than writing a large function to iterate the new record fieldwise against the schema?
评论 #32180844 未加载
BrandiATMuhkuh将近 3 年前
This is really cool. This will make it much simpler to convert from firestore to supabase. I think that&#x27;s the last missing feature that firestore provided which supabase didn&#x27;t.<p>We are already running a sync process between firestore and postgres. So we can do aggregations on JSON data. At this point it&#x27;s only a matter of time before we move to superbase
mdaniel将近 3 年前
It would be valuable to know <i>which</i> JSON-Schema it supports, since there are currently 4 different versions that differ in their capabilities (as one might expect). Related to that, does it honor the &quot;$schema&quot; key allowing the schema to declare which version it is?<p>The postgres-json-schema alternative that&#x27;s mentioned in the repo also ships with what appears to be a conformance test suite; does this carry the same, or was the focus more on speed?
评论 #32181159 未加载
评论 #32181125 未加载
edmundsauto将近 3 年前
I have a very dumb question: why would you use this instead of a traditional schema? I thought the value of json columns was to be partially schemaless and flexible<p>What is the use case for this versus normal column definitions, if you’re looking to enforce schemas?
评论 #32184177 未加载
评论 #32190138 未加载
评论 #32186072 未加载
dividuum将近 3 年前
Not experienced with Postgres and its ecosystem unfortunately, but all those Postgres extensions popping up on hn lately certainly make me envious. To someone with more insight: How risky is it to rely on those extensions? I guess rust handles the &#x27;accidental data corruption or crashes&#x27; aspect. How difficult is it to continue to use such an extension once the original author walks away? Is the extension API somewhat (or perfectly?) stable? Given that this example probably mostly used in CHECK contraints, I guess it could be fairly easy removed or replaced from a running installation?
评论 #32182934 未加载
cpursley将近 3 年前
I just love the work both Supabase &amp; Hasura have done making people aware of how powerful Postgres is.
shroompasta将近 3 年前
How does this validate data with a variable amount of keys with the same value type for example a to-do list<p>my day to day to do list varies in the number of tasks, but the completion will always be in boolean<p><pre><code> [ { &quot;task&quot;: &quot;do Foo&quot;, &quot;completed&quot;: False, }, { &quot;task&quot;: &quot;do Bar&quot;, &quot;completed&quot;: False, }, { &quot;task&quot;: &quot;do Baz&quot;, &quot;completed&quot;: False, }, ... ] </code></pre> Also, what is the issue of schema validation before inserting into the json column, as this is what I&#x27;m doing with a small microservice with Redis.
tsujp将近 3 年前
This is awesome! I remember playing with Hasura (which uses Postgres) a few years ago. Hasura is great but one of the things I really wanted was JSON schema validation at a database level so I could keep all the &quot;raw&quot; types together if that makes sense. I was then, and do now, do public-facing schema validation but that doesn&#x27;t necessarily validate how it&#x27;s going to be persisted into the database so being able to have that closer to the database level now is great.
mekoka将近 3 年前
As an aside, I&#x27;m a long time backend developer writing my first mobile app with Dart&#x2F;Flutter. I tried the popular backend recommendation in that ecosystem. After weeks wasted on it, I googled <i>&quot;{PopularBackend} alternatives&quot;</i> out of frustration and this thing called &quot;Supabase&quot; appeared in the results. What a breath of fresh air it&#x27;s been. It uses Postgres as a backend (with postgREST), which means I can put all those skills to good use (you can go far with all the batteries Postgres comes equipped with: row-level security, triggers, functions, etc). It&#x27;s open source and the free tier lets me do pretty much all I need for development (and probably for a production MVP). Which means I don&#x27;t have to worry about &quot;after questions&quot; until after.<p>Supabase team keep doing what you&#x27;re doing!
评论 #32184140 未加载
klysm将近 3 年前
I was just looking into this for handling typescript integration with postgres. I think there&#x27;s a lot of opportunity to make that work really well. Zapatos and pg-typegen are good steps but it could be even better.
warmwaffles将近 3 年前
If adding this check to an existing table with millions of records, will it scan the entire table checking all of the records, or just check when the records are inserted or updated.
评论 #32180889 未加载
pella将近 3 年前
Thank you!<p>Some suggestion for the next roadmap:<p>- a Dockerfile ( The dockerfile helps me a lot in trying out new technologies )<p>- info about the compatibility with new PG15<p>- CI&#x2F;CD
评论 #32180744 未加载
legalcorrection将近 3 年前
So we’ve come full circle and now JSON is just XML with lighter syntax.
评论 #32181115 未加载
评论 #32180861 未加载
评论 #32180810 未加载
评论 #32181163 未加载