Faster Protocol Buffers (2019)

My go-to for needing to deserialize structured data in a fast way these days is flatbuffers[1]. It compacts nicely and more importantly is zero copy&#x2F;allocation(within the constraints of your language where possible) in deserialize. Which lets you do neat things like mmap it from disk.<p>We used to store 20-30mb of animation data with it and we&#x27;d just mmap the whole file and let the kernel handle paging it in&#x2F;out, worked great.<p>I don&#x27;t know how up to date their benchmarks[2] are but my experience has been that it beats almost every other off-the-shelf solution(other than maybe capn-proto which has some similar properties).<p>[1] <a href="https:&#x2F;&#x2F;google.github.io&#x2F;flatbuffers&#x2F;" rel="nofollow">https:&#x2F;&#x2F;google.github.io&#x2F;flatbuffers&#x2F;</a><p>[2] <a href="https:&#x2F;&#x2F;google.github.io&#x2F;flatbuffers&#x2F;flatbuffers_benchmarks.html" rel="nofollow">https:&#x2F;&#x2F;google.github.io&#x2F;flatbuffers&#x2F;flatbuffers_benchmarks....</a>

Article author here, good to see it on HN, someone else has submitted it (thanks :-)).<p>If you are interested in the topic you may be also interested in a research library I wrote recently: <a href="https:&#x2F;&#x2F;github.com&#x2F;splunk&#x2F;exp-lazyproto" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;splunk&#x2F;exp-lazyproto</a>, which among other things exploits the partial (de)serialization technique. This is just a prototype for now, one day I may actually do a production quality implementation.

One option not mentioned in this article is use of <i>unknown fields</i> [1] (e.g. C++ documentation is at [2]). Rather than changing the types of the field from a message to bytes, simply remove them altogether:<p><pre><code> message Metric { &#x2F;&#x2F; MetricDescriptor metric_descriptor = 1; &#x2F;&#x2F; Resource resource = 2; repeated Int64TimeSeries int64_timeseries = 3; } </code></pre> When then message is deserialised, those will be placed in a separate binary buffer, which is used directly when reserialising. It even works for other types of field. Originally this functionality was lost in the transition from proto2 to proto3 but was added back a long time ago now.<p>This feels like a cleaner solution to me as it&#x27;s what this feature is really intended for.<p>[1] <a href="https:&#x2F;&#x2F;developers.google.com&#x2F;protocol-buffers&#x2F;docs&#x2F;proto3#unknowns" rel="nofollow">https:&#x2F;&#x2F;developers.google.com&#x2F;protocol-buffers&#x2F;docs&#x2F;proto3#u...</a><p>[2] <a href="https:&#x2F;&#x2F;developers.google.com&#x2F;protocol-buffers&#x2F;docs&#x2F;reference&#x2F;cpp&#x2F;google.protobuf.unknown_field_set" rel="nofollow">https:&#x2F;&#x2F;developers.google.com&#x2F;protocol-buffers&#x2F;docs&#x2F;referenc...</a>

Did the OP consider sending less data? For example, registering the metric and using an int, rather than sending all of the static data every time. I guess the major downside there is it would make the metric server stateful and I think OpenTelemetry uses UDP.

The partial decoding is also interesting for another case, I think: polymorphism. The OneOf implementation in go protobuf is ugly, but this trick could make it bareable. I&#x27;ll have to explore further :)

In this case it is also possible to encode the message up to and including fields 1 and 2, then repeatedly encoding field 3 and concatenating the resulting buffers. Protobuf is designed to do this.