TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

NoProto: Flexible, Fast and Compact Serialization with RPC

36 pointsby excerionsforteover 3 years ago

8 comments

spenczar5over 3 years ago
This seems pretty confused. The &quot;compiled vs dynamic&quot; distinction is a property of the <i>implementation</i>, not of the protocol.<p>For example, you can certainly compile Avro into Go source files [0]. You can even compile Avro loaded schemas _during runtime_ into Python bytecode, since Python is interpreted [1]. This even works if you have the _wrong schema document_ for the message (you&#x27;ll just get the subset of fields which are accurately described), because of Avro&#x27;s schema compatibility rules.<p>Likewise, you can deserialize arbitrary protobuf messages during runtime without a compilation step, if you have a description for the message schema. The Python protobuf library has had a &quot;ParseMessage&quot; API forever, and protoreflect [2] exists for Go. (In case it&#x27;s not obvious, I mostly work in Python and Go but I am completely certain analogues exist in other major languages).<p>There is a very big and important difference between a protocol and the <i>implementation</i> of a protocol. I think this README&#x27;s author is not clear on that difference, which shows up in other claims (&quot;Deserialization is incrimental&quot;, for example) too.<p>---<p>[0] <a href="https:&#x2F;&#x2F;github.com&#x2F;actgardner&#x2F;gogen-avro" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;actgardner&#x2F;gogen-avro</a><p>[1] <a href="https:&#x2F;&#x2F;github.com&#x2F;spenczar&#x2F;avroc" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;spenczar&#x2F;avroc</a><p>[2] <a href="https:&#x2F;&#x2F;pkg.go.dev&#x2F;google.golang.org&#x2F;protobuf&#x2F;reflect&#x2F;protoreflect" rel="nofollow">https:&#x2F;&#x2F;pkg.go.dev&#x2F;google.golang.org&#x2F;protobuf&#x2F;reflect&#x2F;protor...</a>
评论 #28411526 未加载
vlovich123over 3 years ago
Does anyone know what the `mutable` column means? I think the cap&#x27;n&#x27;proto section may not be correct as I think you can do read&#x2F;modify&#x2F;write.<p>&gt; all the buffers&#x2F;data is trusted<p>This part is also wrong I think. Cap&#x27;n&#x27;proto is fine to use with untrusted data AFAIK.<p>&gt; Thus, Cap’n Proto checks the structural integrity of the message just like any other serialization protocol would. And, just like any other protocol, it is up to the app to check the validity of the content.<p>Likely the author is referring to this section of the home page:<p>&gt; As of this writing, Cap’n Proto has not undergone a security review, therefore we suggest caution when handling messages from untrusted sources<p>Which seems more like a disclaimer than &quot;you can&#x27;t do this&quot; (security is on by default as compared with FlatBuffers where it&#x27;s opt-in).
评论 #28411041 未加载
erik_seabergover 3 years ago
Being able to quickly replace or append a field without re-serializing the entire struct can be useful, but the overhead for it is pretty substantial (an extra u32 for every field, plus 1&#x2F;4 of another u32 for next vtable in the list). Every u32 always consumes four (unaligned?) bytes. I think “far more space efficient” than Avro is pretty unlikely.<p>I hope he’s also looking at ASN.1 PER for more “don’t write anything the reader already knows” ideas in the vein of Avro.<p>Without OIDs or UUIDs for versions of types, most of these formats seem equally prone to treating invalid messages as unpredictable gibberish or erroring out on an invalid size or offset.
pmalyninover 3 years ago
Tbh it’s kind of funny folks keep reinventing the wheel when we have had portable, standardized serialization format for around 50 years: DER.
评论 #28412802 未加载
评论 #28414456 未加载
Groxxover 3 years ago
Every popular RPC format I&#x27;m aware of supports dynamic schemas, they&#x27;re just generally not used unless strictly necessary. E.g. Cap&#x27;n Proto has SchemaLoader (dynamically loading compiled schemas) and SchemaParser (parsing schema strings into generic objects): <a href="https:&#x2F;&#x2F;capnproto.org&#x2F;cxx.html#dynamic-reflection" rel="nofollow">https:&#x2F;&#x2F;capnproto.org&#x2F;cxx.html#dynamic-reflection</a>, protobuf has &quot;Dynamic Message&quot;: <a href="https:&#x2F;&#x2F;developers.google.com&#x2F;protocol-buffers&#x2F;docs&#x2F;reference&#x2F;cpp&#x2F;google.protobuf.dynamic_message" rel="nofollow">https:&#x2F;&#x2F;developers.google.com&#x2F;protocol-buffers&#x2F;docs&#x2F;referenc...</a> , etc. Plus, literally anything you can compile can be done dynamically, so it&#x27;s not like this is an inherent quality of <i>an encoding</i>, only at best <i>a specific tool</i>.<p>Broadly: I&#x27;m sorta curious about the details here, but I&#x27;ll have to read more later. It seems to be claiming the world without a whole lot of evidence, and some incorrect claims, while being implemented as a fairly simple (often good!) serialized form which is also quite large (less good!). Also I have no idea why it&#x27;s so obsessed with sort-ability. It&#x27;s not like the encoded data as a whole can be meaningfully sorted, and per-value that sounds like &quot;is big-endian&quot;...<p>E.g. this is the serialized form of a pre-defined struct with a single field, like `{&quot;age&quot;:20}`: <a href="https:&#x2F;&#x2F;docs.rs&#x2F;no_proto&#x2F;0.9.60&#x2F;no_proto&#x2F;format&#x2F;index.html" rel="nofollow">https:&#x2F;&#x2F;docs.rs&#x2F;no_proto&#x2F;0.9.60&#x2F;no_proto&#x2F;format&#x2F;index.html</a><p><pre><code> &#x2F;&#x2F; [0, 0, 0, 0, 0, 6, 0, 0, 0, 26, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 20] &#x2F;&#x2F; [ root ptr, vtable, data] </code></pre> which is pretty far from protocol buffers&#x27;: <a href="https:&#x2F;&#x2F;developers.google.com&#x2F;protocol-buffers&#x2F;docs&#x2F;encoding" rel="nofollow">https:&#x2F;&#x2F;developers.google.com&#x2F;protocol-buffers&#x2F;docs&#x2F;encoding</a><p><pre><code> 08 96 20 ^-- 01 in sample </code></pre> This sort of waste in favor of simplicity seems to be spread throughout the encoding. I&#x27;ll happily admit that I think NoProto&#x27;s format is the easier of the two to read and understand by hand, but that&#x27;s not generally why we choose binary encodings. It&#x27;s possible the simplicity gives it a performance edge, but I strongly suspect that, in practice, that&#x27;s influenced far more by the various implementations&#x27; internal details than the encoding itself.
gravypodover 3 years ago
One of the big benefits of protos is that you have a language-agnostic schema that can evolve safely and backwards compatibly.<p>I&#x27;d love for some team to join that with better performance as it seems like it&#x27;s possible from what this project has done.
tomberekover 3 years ago
At what point do you throw in the towel and just send in-memory representations of SQLite DBs around?
omegalulwover 3 years ago
How is serialization and de-serialization faster than compiled formats with just spend? If you update many if the fields some more than once, the overhead would be higher?