Looking at the source code, this seems to work by generating dedicated parser code for a yiven definition which will copy values in a certain order through a flat copy.<p>I'm seeing little specifications or conversions regarding endianness so I'm guessing that's out of scope for this project. It seems almost completely backwards incompatible and I'm not too sure about their security validations. I don't think this and Flatbuffers are competing in the same space, really.<p>I definitely believe this is fast, it's as close to a memcpy to a network packet as you can get. I'd be wary to use this on external data in any native language without any kind of fuzzing first.<p>That said, I do like the way the generators work.
> Karmem has proven to be ten times faster than Google Flatbuffers<p>I’d recommend not using the word “proven” here. In computer science this word typically refers to a mathematical proof. In this case it seems that you ran a regular benchmark for some schemas.<p>I’d also like to see more what the benchmark actually <i>does</i>. A typical trade-off of these formats is how much you do up-front vs on-demand. E.g. accessing fields after multiple variable-length field: Here it’s possible during “decoding” to make sure all fields can be accessed in O(1), or you can do nothing and then every time you access a field you compute the field location. Whether the benchmark accesses the field once or ten times will make a huge difference.<p>In general: If you’re just telling me that it’s 10 times faster without explaining <i>why</i> I will be skeptical.
It's all trade-offs.<p>Flatbuffers trades off encoding speed, programmer ergonomics and binary size (it produces many bytes and it's awkward and still pretty slow to encode) for decoding speed (almost a no-op if you forego buffer verification, which you shouldn't most of the time). Imho it's not a good choice for network wire formats, but for storage it's pretty good.
Go never really clicked with me, but isn't the point of serialization formats interoperability?<p>Like, ok, its 10x faster unzipping than another obscure language dependent format, but how is that better than perl storables or python pickles or ruby ser's other than being "faster"?<p>How do i call this from java or dotNet, and why would i do this other than to make everyone I work with miserable to adopt yet another format?
To get accepted in most of the game engines, the author would need to provide a way to override malloc/realloc/free - even better if no need to realloc.
That is an impressive performance claim, almost 10 times faster than flatbuffers.<p>Where is the flatbuffers native C (or C++) implementation of the benchmark? Are memory allocations avoided/excluded in the benchmark?
Don't know if the owner will ever read this comment, but please add some sections on:<p><pre><code> * Its design goals and rationale
* How those decisions are translated into the actual performance
* What is the trade off made to achieve that
* Why should/shouldn't anyone else use it
</code></pre>
Rather than just a vague performance claim that it's ten times faster than something else. It's not just for this specific library, but applicable to any libraries seeking for broader audiences.
There one commit referencing my favorite data structure [1], the discriminated union (DU) / tagged union / enums with values:<p>> kmparser: implement id generator<p>> That is the first step to implement Unions/Interfaces, it's also useful to
know what is the expected message type to decode.<p>I don't see any other mention or plan about DU's in the repo or metadata. I'm curious what their position is on it.<p>[1]: <a href="https://github.com/inkeliz/karmem/commit/626e6d3b380eb5236c9a240978b1451662cb24d9" rel="nofollow">https://github.com/inkeliz/karmem/commit/626e6d3b380eb5236c9...</a>
I suspect a lot of the speed comes from structure specific serialization (avoiding reflect). This can probably done with less unsafe code, and for most use cases that'd be a better trade-off.
What’s the backwards compatibility story for coding using Karmem? When is it legal to add, modify, or remove a struct field without having to recompile all of the binaries using this format and replace them atomically? When is it legal to add, modify, or remove a struct field without requiring code to be refactored? What about enum variants?<p>These questions may not matter for every use case (e.g. you ship a single binary from a single codebase) but I think that clearly defining these rules opens up a lot of very cool use cases that are otherwise prohibited.
Keeping some context in mind is probably helpful here. The target is WASM. And if you look at the organization the repo own is a part of, it is a web wallet for the cryptocurrency Nano.<p>So perhaps using a generic message serialization library is too slow for its use case since WASM's data types are just ints and floats since the parsing code can't behave like on a native CPU with things like bytes and C-structs?<p>It would have been great if they had disclosed links to issues regarding out-of-bounds access for things like Protobuf or Flatbuffer.
rkyv and postcard seem to be very promising and have been in development for a little while now<p><a href="https://rkyv.org/" rel="nofollow">https://rkyv.org/</a>
<a href="https://github.com/jamesmunns/postcard" rel="nofollow">https://github.com/jamesmunns/postcard</a><p>postcard seems like it would be particularly strong for the wasm use case as it produces small messages that are light in memory.
The (admittedly self-reported, but by <i>fucking Google</i>) FlatBuffers benchmarks are here: <a href="https://google.github.io/flatbuffers/flatbuffers_benchmarks.html" rel="nofollow">https://google.github.io/flatbuffers/flatbuffers_benchmarks....</a>.<p>My anecdotal experience ties out with those FWIW.<p>10x "faster" than that is something targeting an FPGA, and I don't see any Verilog in the repo.<p>Come on folks, #1?