Hi all, Cap'n Proto author here. Thanks for the post.<p>Just wanted to note that although Cap'n Proto hasn't had a blog post or official release in a while, development is active as part of the Sandstorm project (<a href="https://sandstorm.io" rel="nofollow">https://sandstorm.io</a>). Cap'n Proto -- including the RPC system -- is used extensively in Sandstorm. Sandboxed Sandstorm apps in fact do all their communications with the outside world through a single Cap'n Proto socket (but compatibility layers on top of this allow apps to expose an HTTP server).<p>Unfortunately I've fallen behind on doing official releases, in part because an official release means I need to test against Windows, Mac, and other "supported platforms", whereas Sandstorm only cares about Linux. Windows is especially problematic since MSVC's C++11 support is spotty (or was last I tried), so there's usually a lot of work to do to get it working.<p>As a result Sandstorm has been building against Cap'n Proto's master branch so that we can make changes as needed for Sandstorm.<p>I'm hoping to get some time in the next few months to go back and do a new release.
For some reason, the banner (infinitely faster?), name, and introductory FAQ-style responses made me think the whole thing is a joke - similar to Vanilla JS [1].<p>Anyways, it seems like a cool project, so I'll be sure to follow its development closely.<p>[1]: <a href="http://vanilla-js.com/" rel="nofollow">http://vanilla-js.com/</a>
I've always liked Cap'n Proto because it was (quite literally) the ideas behind Protobuf taken to an extreme, or, depending on your point-of-view, reduced to its most basic components: data structures already have to sit in memory looking a certain way, why can't we just squirt that on the wire instead of some fancy bespoke type-length-value struct?<p>Of course, the hardest part is convincing everyone that it's not <i>your bespoke</i> type-length-value struct, but that you have <i>good reasons</i> for what you're doing. I think the humorous, not-so-self-serious presentation has worked in its favor (but that's just a subjective opinion and I can't back it up with data).
To get an overview of the area of binary interchange formats that are language agnostic, the author of Cap'n Proto does a good job in this:<p><a href="https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-sbe.html" rel="nofollow">https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-...</a><p>"Protocol Buffers" has been the go-to for a long time but there are more options now.<p>For uses where serialization/deserialization CPU time is a concern, it seems to really a question of Cap'n Proto versus flatbuffers ( <a href="https://google.github.io/flatbuffers/" rel="nofollow">https://google.github.io/flatbuffers/</a> ).
Big fan of what Sandstorm is doing, both with Sandstorm itself and this component. I <i>really</i> want to use this instead of gRPC, as it seems technically superior, but language bindings and adoption across language ecosystems are likely to be a big downside given that (as Kenton mentions in a comment elsewhere here) Sandstorm isn't really interested in Cap'n Proto being widely adopted. All my new stuff is built in Rust, so the Sandstorm team's interest in and use of Rust are a good fit for me. But when it comes to interoperability with other languages, this may end up being a concern compared to gRPC. In any case, I hope to see the Rust implementation eventually replace the C++ one as the official reference implementation.
Correct me if I'm wrong, but some of this sounds like <i>blitting</i>, except optimized for the in-memory structure and not the on-disk structure.<p>In 2008, Joel Spolsky wrote about 1990s-era Excel file formats and how they used this technique to deal with how slow computers were then [1]. Same technique, new problem set.<p>[1] <a href="http://www.joelonsoftware.com/items/2008/02/19.html" rel="nofollow">http://www.joelonsoftware.com/items/2008/02/19.html</a>
Fractalide (<a href="http://githib.com/fractalide/fractalide" rel="nofollow">http://githib.com/fractalide/fractalide</a>) is an implementation of dataflow programming (specifically flow based programming). Component build hierarchies are coordinated via the Nix package manager. Capnproto contracts are weaved into each component just before build time. These contracts are the only way compenents talk to each other. Thanks Sandstorm.io for this great software.
I've always liked the look of this, saw it a while back but we are still using protobuf in our .NET environment simply due to the "free" schema generation using AOP/attributes [ProtoContract]/[ProtoMember] in Marc Gravell's excellent protobuf-net (<a href="https://github.com/mgravell/protobuf-net" rel="nofollow">https://github.com/mgravell/protobuf-net</a>) project - I assume this would also be possible for cap'n proto.
Could this be used as an alternative to Apache Arrow[1]?<p>[1] <a href="https://arrow.apache.org/" rel="nofollow">https://arrow.apache.org/</a>
I hate to ask but can anyone explain this like I'm from 2000?<p>I guess it's a way to send data to your front end java script but not use json and this compresses it so it's faster? How much better than using json is it?
> The Cap’n Proto encoding is appropriate both as a data interchange format and an in-memory representation, so once your structure is built, you can simply write the bytes straight out to disk!<p>Eh, I'd rather pay the cost of serialisation once and deserialisation once, and then access my data for as close to free as possible, rather than relying on a compiler to actually inline calls properly.<p>> Integers use little-endian byte order because most CPUs are little-endian, and even big-endian CPUs usually have instructions for reading little-endian data.<p><i>sob</i> There are a lot of things Intel has to account for, and frankly little-endian byte order isn't the worst of them, but it's pretty rotten. Writing 'EFCDAB8967452301' for 0x0123456789ABCDEF is perverse in the extreme. Why? <i>Why?</i><p>As pragmatic design choices go, Cap'n Proto's is a good one (although it violates the standard network byte order). Intel effectively won the CPU war, and we'll never be free of the little-endian plague.<p>It's all so depressing.
How do you pronounce the name? If libreoffice is bad.. This name is absolutely impossible.<p>Is it captain? Is it cap+n+proto?<p>A lot of collaboration is verbal - people sit around and talk about stuff. I don't know if it is a fun take on an American word... But it is impossible to use in the rest of the world.<p>I really wish you would call it something else... Unless it is personal for you :(
Here's the discussion from a while ago: <a href="https://news.ycombinator.com/item?id=5482081" rel="nofollow">https://news.ycombinator.com/item?id=5482081</a>
I think that compression is a must for the serialization library. Protobuf uses almost twice less memory than Cap'n Proto. Using an external compression is not an option in some cases. E.g. consider building tcp-server that communicates with thousands of clients simultaneously. Each client connection will have its own LZ4 context that should be heap allocated. I believe it's about 16KB per connection + buffers. This results in large memory consumption and a lot of random memory access and TLB misses.
Interfaces! Inheritance! Looks promising. Protocol buffers are nice for their compact encoding and multi-language generator support but as a schema language they are really cumbersome. Composition is pretty much all you get, there are no longer required fields, you can't even use enums as a key type in a map. I'm sure their use cases are not necessarily the same as mine but sometimes I miss just using plain old XML.
> capability-based RPC system.<p>This sounds like a cool idea, but so far I haven't seen any good explanation of how it works, and why it will save me from rolling my own ACL system. For bragging about it in the very first sentence, there is surprisingly little detail about how it works.