I played quite a bit with MessagePack, used it for various things, and I don't like it. My primary gripes are:<p>+ The Object and Array needs to be entirely and deep parsed. You cannot skip them.<p>+ Object and Array cannot be streamed when writing. They require a 'count' at the beginning, and since the 'count' size can vary in number of bytes, you can't even "walk back" and update it. It would have been MUCH, MUCH better to have a "begin" and "end" tag --- err pretty much like JSON has, really.<p>You can alleviate the problems by using extensions, store a byte count to skip etc etc but really, if you start there, might as well use another format altogether.<p>Also, from my tests, it is not particularly more compact, unless again you spend some time and add a hash table for keys and embed that -- but then again, at that point where it becomes valuable, might as well gzip the JSON!<p>So in the end it is a lot better in my experience to use some sort of 'extended' JSON format, with the idiocies removed (trailing commas, forcing double-quote for keys etc).
Although MessagePack is definitely <i>not</i> a drop-in replacement for JSON, it is certainly extremely useful.<p>Unlike JSON, you can’t just open a MessagePack file in Notepad or vim and have it make sense. It’s often not human readable. So using MessagePack to store config files probably isn’t a good idea if you or your users will ever need to read them for debugging purposes.<p>But as a format for something like IPC or high-performance, low-latency communication in general, MessagePack brings serious improvements over JSON.<p>I recently had to build an inference server that needed to be able to communicate with an API server with minimal latency.<p>I started with gRPC and protobuf since it’s what everyone recommends, yet after a lot of benchmarking, I found a way faster method to be serving MessagePack over HTTP with a Litestar Python server (it’s much faster than FastAPI), using msgspec for super fast MessagePack encoding and ormsgpack for super fast decoding.<p>Not sure how this beat protobuf and gRPC but it did. Perhaps the Python implementation is just slow. It was still faster than JSON over HTTP, however.
CBOR: It’s like JSON but fast and small but also an official IETF standard.<p><a href="https://cbor.io/" rel="nofollow">https://cbor.io/</a>
MessagePack saves a little bit of space and CPU ... but not a lot:<p><a href="https://media.licdn.com/dms/image/v2/D5612AQF-nFt1cYZhKg/article-inline_image-shrink_1000_1488/article-inline_image-shrink_1000_1488/0/1703965440210?e=2147483647&v=beta&t=XOmhYJP5X6iYJ17RYsk3SjX9FTvfcZbQFP-OTOg6flk" rel="nofollow">https://media.licdn.com/dms/image/v2/D5612AQF-nFt1cYZhKg/art...</a><p>Source: <a href="https://www.linkedin.com/pulse/json-vs-messagepack-battle-data-efficiency-akshay-singh-kanawat-7dx2c" rel="nofollow">https://www.linkedin.com/pulse/json-vs-messagepack-battle-da...</a>
In my experience protobuf was smaller than MessagePack. I even tried compressing both with zstd and protobuf was still smaller. On the other hand protobuf is a lot less flexible.
I discovered JSON Binpack recently, which works either schemaless (like msgpack) or - supposedly more efficiently - with a schema. I haven't tried the codebase yet but it looks interesting.<p><a href="https://jsonbinpack.sourcemeta.com/" rel="nofollow">https://jsonbinpack.sourcemeta.com/</a>
Does it solve the problem of repeating set of keys in an object array, eg. when representing a table?<p>I don't think using a dictionary of key values is the way to go here. I think there should be a dedicated "table" type, where the column keys are only defined once, and not repeated for every single row.
It drops the most useful aspect of JSON, which is that you can open it in a text editor.<p>It's like JSON in that it's a serialisation format.
Ignorant question - is the relatively small size benefit worth another standard that's fairly opaque to troubleshooting and loses readability?<p>Is there a direct comparison of why someone should choose this over alternatives? 27 bytes down to 18 bytes (for their example) just doesn't seem like enough of a benefit. This clearly isn't targeted to me in either case, but for someone without much knowledge of the space, it seems like a solution in search of a problem.
I really like MessagePack as a serialization format but it does leave a lot underspecified, such as how to handle decoding/encoding integers. While the encoding has signed and unsigned types, the data model does not. Its not straightforward to translate to a data midel with separate signed and unsigned types.<p>The separation between str and bin types doesnt make sense for my usecases, and others with similar usecases either always use bin or always use str. logically bin would make sense, but str has fixstr and theres no fixbin, so more efficient to use str. Both are valid but thats more discrepancies to deal with.<p>For my personal usecases I'm thinking of forking MessagePack, removing str and ext, reallocting tag space a bit, having a clear mapping to low level language types, adding 128 bit integers, and maybe a frame type for streams.
A trick I often use to get the most out of messagepack is using array encoding of structs. Most msgp libraries have support for this.<p>This gives you a key benefit of protobuf, without needing external schema files: you don't have to pay the space of the keys of your data.<p>This is simply not something you can do with JSON, and depending on your data can yield substantial space savings.
In our C++ project, we use the nlohmann library for handling JSON data. When sending JSON over MQTT, we leverage a built-in function of nlohmann to serialize our JSON objects into the MessagePack format. You can simply call jsonObj.to_msgpack(). Similarly, the data can be decoded back easily.
ZigPack is a pretty good MessagePack implementation written in Zig: <a href="https://github.com/thislight/zigpak">https://github.com/thislight/zigpak</a>