Demystifying the protobuf wire format

"Demystifying" is a big word for what the original docs document quite well, and is also not like you couldn't read and understand that in few hours, if you are not totally foreign to protocol design and serialization? This post gives even much less information?!

I did a lot of research on binary serialization at the University of Oxford. One of the papers I published is a comprehensive review of existing JSON-compatible serialization formats (<a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2201.02089" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2201.02089</a>). It touches on Protocol Buffers (and &gt;10 other formats) and I&#x27;m analyzing the resulting hexadecimals close to how the OP is doing.<p>I also published a space-efficiency benchmark of those same formats (<a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2201.03051" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2201.03051</a>) and ended up creating <a href="https:&#x2F;&#x2F;jsonbinpack.sourcemeta.com" rel="nofollow">https:&#x2F;&#x2F;jsonbinpack.sourcemeta.com</a> as a proposed technology that does binary serialization of JSON using JSON Schema.

As a counterpoint to the horror stories, I&#x27;ve had a few relatively good experiences with protocol buffers (not gRPC). On one project, we had messages that needed to be used across multiple applications, on a microcontroller, on an SBC running Python, in an Android app, in a web service, and on web UI frontend. Being able to update a message definition in one place, and have it spit out updated code in half a dozen languages while allowing for incremental rollout to the various pieces was <i>very handy</i>.<p>Sure - it wasn&#x27;t all guns and roses, but overall it rocked.

We built a backend heavily using protobufs&#x2F;grpc and I highly regret it.<p>It ads an extra layer of complexity most people don&#x27;t need.<p>You need to compile the protobufs and update all services that use them.<p>It&#x27;s extra software for security scans.<p>Regular old http 1 rest calls should be the default.<p>If you are having scaling problem only then should you consider moving to grpc.<p>And even then I would first consider other simpler options.

It was also used for Farsight&#x27;s tunnelled SIE called NMSG. I wrote a pure python protobuf dissector implementation for use with Scapy (<a href="https:&#x2F;&#x2F;scapy.readthedocs.io&#x2F;en&#x2F;latest&#x2F;introduction.html" rel="nofollow">https:&#x2F;&#x2F;scapy.readthedocs.io&#x2F;en&#x2F;latest&#x2F;introduction.html</a>) for dissecting &#x2F; tasting random protobuf traffic. I packaged it with an NMSG definition (<a href="https:&#x2F;&#x2F;github.com&#x2F;m3047&#x2F;tahoma_nmsg">https:&#x2F;&#x2F;github.com&#x2F;m3047&#x2F;tahoma_nmsg</a>).<p>I re-used the dissector for my Dnstap fu, which has since been refactored to a simple composable agent (<a href="https:&#x2F;&#x2F;github.com&#x2F;m3047&#x2F;shodohflo&#x2F;tree&#x2F;master&#x2F;agents">https:&#x2F;&#x2F;github.com&#x2F;m3047&#x2F;shodohflo&#x2F;tree&#x2F;master&#x2F;agents</a>) based on what was originally a demo program (<a href="https:&#x2F;&#x2F;github.com&#x2F;m3047&#x2F;shodohflo&#x2F;blob&#x2F;master&#x2F;examples&#x2F;dnstap2json.py">https:&#x2F;&#x2F;github.com&#x2F;m3047&#x2F;shodohflo&#x2F;blob&#x2F;master&#x2F;examples&#x2F;dnst...</a>) because &quot;the people have spoken&quot;.<p>Notice that the demo program (and by extension dnstap_agent) convert protobuf to JSON: the demo program is &quot;dnstap2json&quot;. It&#x27;s puzzlingly shortsighted to me that the BIND implementation is not network aware it only outputs to files or unix sockets.<p>The moment I start thinking about network traffic &#x2F; messaging the first question in my mind is &quot;network or application&quot;, or &quot;datagram or stream&quot;? DNS data is emblematic of this in the sense that the protocol itself supports both datagrams and streams, recognizing that there are different use cases for distributed key-value store. JSON seems punctuation and metadata-heavy for very large amounts of streaming data, but a lot of use cases for DNS data only need a few fields of the DNS request or response so in practice cherry picking fields to pack into a JSON datagram works for a lot of classes of problems. In my experience protobuf suffers from a lack of &quot;living off the land&quot; options for casual consumption, especially in networked situations.

Why not just use cap’n’proto? It seems superior on every metric and has very impressive vision.<p>Honestly the biggest failing for those guys was not making a good Javascript implementation. Seems C++ aint enough these days. Maybe emcscripten works? Anyone tried it ?<p><a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=25585844">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=25585844</a><p>kenton - if you’re reading this - learn the latest ECMAScript or Typescript and just go for it!

Reddit moved to gRPC and protobuff from Thrift a couple years ago. I wonder how it is going for them. <a href="https:&#x2F;&#x2F;old.reddit.com&#x2F;r&#x2F;RedditEng&#x2F;comments&#x2F;xivl8d&#x2F;leveling_up_reddits_core_the_transition_from&#x2F;" rel="nofollow">https:&#x2F;&#x2F;old.reddit.com&#x2F;r&#x2F;RedditEng&#x2F;comments&#x2F;xivl8d&#x2F;leveling_...</a>

For the ones looking for a minimal and conservative binary format, there is BARE [1]. It is in the process of standardization.<p>[1] <a href="https:&#x2F;&#x2F;baremessages.org&#x2F;" rel="nofollow">https:&#x2F;&#x2F;baremessages.org&#x2F;</a>

I wish DevTools had an API to let extensions display content in the network tab that is something besides JSON or XML. Or add a few things like protobuf.

I thought this was going to be about physically storing memory in wires, i.e. core memory.

Eh, I struggle to say that pb has a &quot;wire&quot; format. A binary encoding sure.<p>To me wire format implies framing etc, enough stuff to actually get it across a stream in a reasonable way. For pb this usually means some sort of length delimited framing you come up with yourself.<p>Similarily pb doesn&#x27;t have a canonical file format for multiple encoded buffers.<p>For these reasons I rarely use pb as an interchange format, it&#x27;s great for internal stuff and good if you want to do your own framing or file format but if you want to store and eventually process things with other things then you are better off with stuff like Avro which does define things like the Object Container Format.

I find it interesting that the folks running away screaming from protobuf are using it in conjunction with gRPC. Is the problem really with the wire format or is it a problem with all of the stuff above?<p>I&#x27;ve been using protobuf for a (non-web) hobbyist project for some time now and find it fairly straightforward to use, especially when working across multiple implementation languages. For me, it seems to be a nice middle-ground between the ease of JSON and the efficiency of a hand-rolled serialization format.

We had a client choose protobufs &#x2F; grpc which totally stalled the developers and created alot of problems and complexity. The client insisted for whatever reason and eventually ran out of money. Their unfinished code is sitting in some Github repository somewhere.<p>Run very fast from it, unless you have a VERY good reason to use it.