Gobs of data (2011)

Interesting. I wonder to what extent it&#x27;s found use at Google over this past decade.<p>There are advantage to being language-specific, but a lot of disadvantages, as well (speaking as someone who recently had to write some Elixir code to unmarshal a Ruby object...). It seems hard to introduce this since you&#x27;re forcing all communicating services to be Go-based, which is kind of contrary to the independence that microservices usually affords you.<p>Some of the benefits are simply design goals (e.g., top level arrays) which could also be done in a language-independent protocol. And even performance questions <i>probably</i> could. Like, Cap&#x27;n Proto I think is designed so that users of the protocol don&#x27;t have to serialize&#x2F;deserialize the data, right? They just pass it around and work with it directly.<p>I can see Rob Pike being frustrated with Protocol Buffers at Google, and I don&#x27;t begrudge anyone for taking a big shot like this, but I wonder if he&#x27;s found any success with it.

Just finished removing this encoding in our production services.<p>It panics on malformed input which is a no go for us since high availability is really important for us, and it showed quite a lot in the performance and memory profiles (roughly 5 times the time and memory as doing the same with JSON).<p>The code was converting some data to gob, and storing it in the database for later.<p>We now just do the same but in json, it’s human readable and Postgres validates that the data is valid JSON.<p>And unmarshaling it does not panic.

&gt;[Required fields are] also a maintenance problem. Over time, one may want to modify the data definition to remove a required field, but that may cause existing clients of the data to crash.<p>Okay, but would you rather have it crash or allow for a program to run on the wrong data? Especially if you do that and then say that everything has zero as a default value.<p>The question remains whether the serialization format should be taking care of that, or a round of parsing later on with a schema on the side; but if you do the former without the latter you&#x27;re setting yourself up for deployment nightmares

Someone made a benchmark of serialization libraries in go [1], and I was surprised to see gobs is one of the slowest ones, specially for decoding. I suspect part of the reason is that the API doesn&#x27;t not allow reusing decoders [2]. From my explorations it seems like both JSON [3], message-pack [4] and CBOR [5] are better alternatives.<p>By the way, in Go there are a like a million JSON encoders because a lot of things in the std library are not really coded for maximum performance but more for easy of usage, it seems. Perhaps this is the right balance for certain things (ex: the http library, see [6]).<p>There are also a bunch of libraries that allow you to modify a JSON file &quot;in place&quot;, without having to fully deserialize into structs (ex: GJSON&#x2F;SJSON [7] [8]). This sounds very convenient and more efficient that fully de&#x2F;serializing if we just need to change the data a little.<p>--<p>1: <a href="https:&#x2F;&#x2F;github.com&#x2F;alecthomas&#x2F;go_serialization_benchmarks">https:&#x2F;&#x2F;github.com&#x2F;alecthomas&#x2F;go_serialization_benchmarks</a><p>2: <a href="https:&#x2F;&#x2F;github.com&#x2F;golang&#x2F;go&#x2F;issues&#x2F;29766#issuecomment-454926474">https:&#x2F;&#x2F;github.com&#x2F;golang&#x2F;go&#x2F;issues&#x2F;29766#issuecomment-45492...</a><p>--<p>3: <a href="https:&#x2F;&#x2F;github.com&#x2F;goccy&#x2F;go-json">https:&#x2F;&#x2F;github.com&#x2F;goccy&#x2F;go-json</a><p>4: <a href="https:&#x2F;&#x2F;github.com&#x2F;vmihailenco&#x2F;msgpack">https:&#x2F;&#x2F;github.com&#x2F;vmihailenco&#x2F;msgpack</a><p>5: <a href="https:&#x2F;&#x2F;github.com&#x2F;fxamacker&#x2F;cbor">https:&#x2F;&#x2F;github.com&#x2F;fxamacker&#x2F;cbor</a><p>--<p>6: <a href="https:&#x2F;&#x2F;github.com&#x2F;valyala&#x2F;fasthttp#faq">https:&#x2F;&#x2F;github.com&#x2F;valyala&#x2F;fasthttp#faq</a><p>--<p>7: <a href="https:&#x2F;&#x2F;github.com&#x2F;tidwall&#x2F;gjson">https:&#x2F;&#x2F;github.com&#x2F;tidwall&#x2F;gjson</a><p>8: <a href="https:&#x2F;&#x2F;github.com&#x2F;tidwall&#x2F;sjson">https:&#x2F;&#x2F;github.com&#x2F;tidwall&#x2F;sjson</a>

Gob is a great serialization format! It&#x27;s super easy to use, and supports go native types (kind of like Python&#x27;s pickle).<p>For a recent project, I needed a simple key-value store. I was evaluating using a full RDBMS, but I ended up just putting gob files in a directory.

FWIW, this isn&#x27;t used much by the community. Being a standard library package it still get some use of course, but for comparison, encoding&#x2F;gob shows about 22.5K imports [1] to encoding&#x2F;json&#x27;s nearly 800K, and whereas you can see in the JSON search an ecosystem of JSON libraries, gob is basically just gob.<p>Calling it &quot;dead&quot; just invites a tedious thread about what the definition of &quot;dead&quot; is, so I won&#x27;t, I&#x27;ll just sort of imply it in this sentence without actually coming out and saying it in a clear manner. I would generally both A: recommend against this, not necessarily as a dire warning, just, you know, a recommendation and B: for anyone who is perturbed by the idea of this existing, just be aware that it&#x27;s not like this package has embedded itself into the Go ecosystem or anything.<p>[1]: <a href="https:&#x2F;&#x2F;pkg.go.dev&#x2F;search?q=gob" rel="nofollow noreferrer">https:&#x2F;&#x2F;pkg.go.dev&#x2F;search?q=gob</a><p>[2]: <a href="https:&#x2F;&#x2F;pkg.go.dev&#x2F;search?q=json" rel="nofollow noreferrer">https:&#x2F;&#x2F;pkg.go.dev&#x2F;search?q=json</a>

Not gobs of comments but discussed at the time:<p><i>Gobs of data</i> - <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=2365430">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=2365430</a> - March 2011 (2 comments)

&gt; If all you want to send is an array of integers, why should you have to put it into a struct first?<p>If you&#x27;re sure that&#x27;s all you&#x27;ll ever have to do, then sure. But unless you&#x27;re 100% certain that the protocol will never evolve further, having a more complex structure allows it to change in a gradual way.

this is quite old, so I&#x27;m curious about what triggered it being posted again, has something happened &#x2F; changed?

I used gob for my first client&#x2F;server Go program, which was a &quot;make one of something you know about to throw away&quot; new language experiment. It worked, but I quickly turned away from it, because it would never be cross platform.<p>I saw gob more as an experiment that the Go team used to check the reflect package&#x27;s usability. (Which sucks anyway, by the way.)<p>I&#x27;m surprised it&#x27;s still in the stdlib. I would have guessed it would have been removed for Go 1.0, because it was already clear then that it was not suitable for anything more experiments.

It may not be a good tool for communicating between services implemented in different languages. But i’d happily use it to save stuff to disk where database is overkill.

I did not know it, but I think so few changes from the proto-buffer that it is a waste of time.

I forgot Go was still around. Thanks for reminding me.