[kentonv]

Hi all, Cap'n Proto author here. Thanks for the post.

Just wanted to note that although Cap'n Proto hasn't had a blog post or official release in a while, development is active as part of the Sandstorm project (https://sandstorm.io). Cap'n Proto -- including the RPC system -- is used extensively in Sandstorm. Sandboxed Sandstorm apps in fact do all their communications with the outside world through a single Cap'n Proto socket (but compatibility layers on top of this allow apps to expose an HTTP server).

Unfortunately I've fallen behind on doing official releases, in part because an official release means I need to test against Windows, Mac, and other "supported platforms", whereas Sandstorm only cares about Linux. Windows is especially problematic since MSVC's C++11 support is spotty (or was last I tried), so there's usually a lot of work to do to get it working.

As a result Sandstorm has been building against Cap'n Proto's master branch so that we can make changes as needed for Sandstorm.

I'm hoping to get some time in the next few months to go back and do a new release.

[StephanTLavavej]

As of MSVC 2015 Update 3, the support for C++11 is nearly complete. The missing pieces are that Expression SFINAE is partially supported (what's there is good enough for the STL and Boost, with certain workarounds that you must be aware of), and C99 preprocessor support is full of bugs. Aside from that, all features are present and generally usable. (Some are buggier than others; e.g. we're overhauling multithreading library support for the next major version of the libraries.)

[Hydraulix989]

There's still a bunch of minor differences/annoyances with the MS toolchain that require me to go in and make a bunch of small changes everywhere to my project, which compiles fine on clang/g++ (and I wouldn't say I'm doing anything _crazy_ like the aforementioned "Expression SFINAE").

Right now, my project's rather small 100 kLOC codebase compiles (with very minimal #ifdef hackery) on Android, iOS, Mac, and Linux; but Windows is still very much a WIP, even some of my third-party libraries don't compile on VC. I'm actually considering trying the MINGW toolchain at this point, and I'd be curious to hear anyone's thoughts?

I'm sure part of it is that the compiler itself probably developed with the tight feedback loop from developing and maintaining large codebases in house at Microsoft (like Windows and Office). It's pretty hard when part of the internal pressure to support large codebases like that conflicts with the need to conform to outside third-party standards. I've heard great things about the compiler people at Microsoft, and I'm sure they have a technically strong team, but they are most likely caught in the middle of this organizational deadlock.

I'm sure it doesn't help that I'm not mentioning any specifics, I'm going to be revisiting the Windows desktop port of my mobile app soon (which I gave up and haven't touched for a couple months), and everything will be fresh in my mind again. I do vaguely remember something about having to explicitly add more #include statements to pull in header files that were already getting pulled in by my other compilers.

[david-given]

I used to develop the Windows port of WordGrinder (my console word processor) using Mingw for Linux, testing with Wine and packaging using nsys for Linux. This actually worked pretty well and allowed a complete Windows development flow from Linux using the same rolling automation for both platforms.

The big problem was Wine, which can behave quite differently to real Windows in some places. It's certainly not bug-for-bug compatible. So I kept finding myself having to boot into Windows anyway for debugging, so eventually I just installed an msys development system on Windows with gvim and used that.

There was also something else wrong with mingw for Linux, but for the life of me I can't remember what. There was something about mingw being different to mingw32? Missing headers? It's been too long.

[networked]

You might mean the differences between MinGW and MinGW-w64. The former is the version hosted at http://mingw.org that used to be called mingw32 (not to be confused with mingw-w64-i686). The latter is what a lot of Linux distributions package today and what you install with MSYS2 [1] on Windows. It was initially forked from MinGW to integrate support for the 64-bit Windows API, hence the name, but it has gained other features since and is more actively maintained.

I've run into a difference between the two recently. It turns out that MinGW implemented POSIX glob() post-fork, which hasn't been ported to MinGW-w64.

[1] As I side note, I found that MSYS2 makes a huge difference in how you develop software for Windows with a POSIXy toolchain. Anyone who uses MinGW on Windows should try it.

[kixpanganiban]

Kenton, thank you for your work on this! I currently use ZeroRPC (which uses protobuf and msgpack) and I was blown away by Cap'n Proto. Really excited to try it out soon! Some questions:

- Do you guys have an RPC library written in anything other than C++? If not, could you point me to protocol specs so I can start writing my own?

- Since it uses a streaming model to support random access, what encryption method do you think would work best with Cap ' n Proto that would keep it speedy and still retain all functionality?

Thanks!

[kentonv]

> - Do you guys have an RPC library written in anything other than C++? If not, could you point me to protocol specs so I can start writing my own?

People have written implementations in Rust, Go, and Erlang, and wrappers around the C++ library in Javascript and Python: https://capnproto.org/otherlang.html

Scroll down that page for some info on how to start writing an implementation in another language.

The RPC protocol spec is here:

https://github.com/sandstorm-io/capnproto/blob/master/c++/sr...

> - Since it uses a streaming model to support random access, what encryption method do you think would work best with Cap ' n Proto that would keep it speedy and still retain all functionality?

Hmm, I'm not clear on what you mean by "streaming model" -- I think of "streaming" as the opposite of random access.

Regarding encryption, this is a very big question and there are a lot of different needs and use cases to consider. Mostly I don't think that use of Cap'n Proto affects encryption decisions much, but if you want to make sure you don't lose random access, you should of course use a cipher that supports random access, like chacha20 or AES-CTR.

[kixpanganiban]

> Hmm, I'm not clear on what you mean by "streaming model" -- I think of "streaming" as the opposite of random access.

Right, sorry, I re-read my comment and confused myself too. Seems like it's bad for me to go on HN without a fresh cup of coffee (it's 10am now here in the Philippines).

Thanks for your swift response, I'll experiment with AES-CTR (since I'm more familiar with it than chacha20). And thanks for pointing out that there are wrappers for Python/Go already, the programming language I use daily and was thinking of building libs for! Again, great work, and I'll stay posted.

[aidenn0]

Also as a note, both ciphers suggested are stream ciphers, so you will need some way of authenticating. Depending on your needs, you may consider using something like libsodium that takes care of all of that for you:

https://download.libsodium.org/libsodium/content/secret-key_...

[jcoffland]

Recently I've started compiling more of my code for Windows using Mingw (or clang). Performance is very good and you can build in Linux for Windows inside a Docker container. You can even build Windows installers using NSIS. I've also tried to do this for OSX but with less success.

[kentonv]

Cap'n Proto supports MinGW as a target platform. (It has also supported Cygwin since very early on.) However, Cap'n Proto is a library, so it needs to support whatever compiler the users of that library are using, and for the vast majority of Windows developers that's MSVC.

[jcoffland]

Right, but you should be able to compile a library with either MinGW or clang from Linux that your users can use with MSVC.

[kentonv]

Not if it uses C++. MinGW uses the G++ ABI which is totally incompatible with the MSVC ABI.

(Also, a lot of the issues we face affect headers, which need to be compiled into the client projects.)

[jcoffland]

Since the code is so small and in C++ why not make it a header only library?

[socmag]

It sounds really well designed. Great job Cap'n Proto, I've been following your work for some time!

I've also built another message format recently (yes I know, they are never ending). It can't do everything Cap'n Proto can do, although it shares a lot of the same values. One thing I chose to do is to have order preservation on types, which can be very useful. It does mean that the wire format is largely BE though. Anyway, that's an aside.

I'm curious what you think of Amazon ION by the way?

Good stuff!

[kentonv]

I haven't looked at Ion before, but it appears to be similar to BSON or Msgpack in that it's a binary format that encodes field names as textual identifiers, to be "self-describing" and avoid the need for an external schema. I generally like schemas, because I like static typing. Of course, static types vs. dynamic types are another ancient flamewar and I'm unlikely to cover any new ground by stating arguments here. :)

[aidenn0]

Schemas are IMO the only way to go when you are transferring between heterogenous languages, even when all languages involved are untyped.

Consider javascript talking to common lisp. Of course JSON has a canonical mapping to javascript, but it does not for common lisp. Should a JS array be a lisp list or vector? Should lisp's NIL be false or null? Should a JS object decode to an alist, plist, or hash-table? &ct.

[nemo1618]

Nitpick: I think you meant "&c". The t is superfluous.

[aidenn0]

Interesting; I've always written &ct, which wikipedia informs me is considered "archaic" Thanks for that note, it will save me one character of typing 3 years from now when I've finally broken the muscle memory of the old way :)

[e12e]

Wikipedia probably mentioned it, but & is a combination of "e" and "t", or "et" - french (or perhaps latin?) for "and". Thus &c is short for "etc", or "et cetera". I wasn't aware of the form "&ct", I presume it stands for "ET CeTera", but I'm not entirely clear on why that extra "T" would ever make sense. I doubt anyone ever used "etct"?

[DonHopkins]

I'm still trying to break the muscle memory of typing &ct ...

[socmag]

I think there is a lot in common between Common Lisp, Python and JavaScript in general.

For many years I was in the schemaless camp before JS came along. Then for a number of years I was in the self-describing camp because I was thinking that if we don't accept JavaScript and JSON are pretty fundamental on the web we're fools and everyone seemed to be passing around JSON. So in that period was thinking that MsgPack was pretty damn good.

Recently I've switched back to the schemaless view, but with strict order preservation and richly typed fields. Very "tuple" based... so works well with Lisp and JavaScript but also C++. Highly inspired by Linda.

I don't do what Cap'n Proto does and lay out the fields and all that good stuff so that you can kind of memory map it onto structs.

That is nice and I understand the motivation for sure, and I have worked on systems that do that in the past with very good results, but currently my thinking is that compactness without additional compression is a good balance.

Also, since the protocol is order preserving (where it matters) you can do radix sort operations or hash maps on the server side extremely quickly. That was the ultimate motivating factor.

[SomeDateTime, "hello", 1.0f, [10,20], "Foo", false, [SomeMatrix]] etc etc...

Inner tuples or BLOB's are length prefixed of course so you can skip them, but basic types and strings are not. Strings are zero terminated while Ints/Floats/Doubles are BE and complement encoded to preserve sorts, and also Integers are packed to minimum size.

Memory mapping is possible on this system too, but it's very "functional"... there's no attempt at pointer preservation. I don't go that far and Cap'n P seems to be preserving some of the semantics of ProtoBuf at least in that regard, which I'm sure is a good thing for many scenarios.

You could still do that with what I'm doing but it would be at the application level. Same with self-description actually, you could easily build something that looks like JSON if you wanted... either:

[["foo",1.0],["boo","cat"]...] etc

or

[[1.0,"cat"],["foo",boo"]

the choice is really up to the application programmer.

That might be a bit "loose" for a lot of people to stomach, but it works very well for what I'm doing, has a lot of flexibility and packs really well

Like I was suggesting, protocols are something of an art and I don't think we're at a final solution yet, which is why many people are constantly inventing new ones :-)

Hopefully we will get to some consensus one day!

[socmag]

Also I might add that I very quickly became quite disenchanted with the lack of types in JSON; and I'm sure many of us have come to that same conclusion.

Really hope that it can be fixed at some point, but not holding my breath on that one.

I think it will take a major effort to reform that format although I am hopeful now that we at least have UInt8Array and friends that are starting to expose a broader set of machine friendly types.

[aidenn0]

I've actually seen people use types with JSON. JSON objects are clearly more than sufficient to represent the data you need, and you can enforce the schema in a library. It's just a much less efficient form of something like protobufs.

[heavenlyhash]

I like schemas for validation and strong types. I also like self-describing systems when all goes to hell and I'm debugging either my own protocols or someone else's.

I have this crazy idea that there's a middle ground: self-describing and schemas?

We can kind of glue this together (there's lots of json schemas floating around out there now, it seems), but it would be awfully interesting to see these well-supported as a pair (with explicitly language agnostic schema definitions -- which seems to be a sticking point for most of the json schema strapons).

</tangent>

[kentonv]

When I'm forced to use JSON, I often do it by defining a Cap'n Proto schema and using capnp::JsonCodec to parse / serialize that schema as JSON.

https://github.com/sandstorm-io/capnproto/blob/master/c++/sr...

(There are also similar libraries for Protobuf.)

[heavenlyhash]

That's..... I want that in my lang.

[DenisM]

http://cbor.io

[heavenlyhash]

Doesn't really have an implementation-agnostic schema spec, as far as I'm aware.

(I use CBOR a lot -- I'm otherwise quite happy with it!)

EDIT: I guess there's a "CDDL" listed on the tools page, but... It's still single implementation (ruby) and I don't see a clear link to a grammar for it.

[socmag]

Hahaha very true let's not go there. Anyway yes I'm a big fan of richly typed schema-less formats as well, even better when they can be read easily by both JS stuff and C++.

I'll definitely give it a whirl. Cheers!

[forrestthewoods]

Treating Windows like a red headed step child is why I, with great disappointment, passed on Cap'n'Proto.

Cross platform libraries that really only care about a single platform is an endless source of frustration. :(

[Twirrim]

Writing software that is safely and gracefully cross-platform is a pain in the neck :-/

[vvanders]

Except in this case you have flatbuffers which is a perfectly reasonable replacement and works across a wide array of platforms.

We rejected Cap'n Proto for the same reasons. I don't see a need to use bleeding edge C++ features when the same(or better) results can be achieved with code generation.

Kudos on the effort but if you're looking for wide adoption you've missed the mark.

[kentonv]

> if you're looking for wide adoption

Well... We're not, actually. We're looking to support the needs of Sandstorm.

It's nice if my code is useful to others, and if people want to contribute better Windows support I'll be totally happy to review and merge those changes! But wide adoption of Cap'n Proto (outside of the Sandstorm context) is not part of our business model, unfortunately.

(Note: The poster you were replying to isn't associated with Cap'n Proto nor Sandstorm.)

[nurettin]

I heard the exact same thing said about templates 15 years ago.

[wvh]

Slightly off-topic but it's good to see after all these years that companies like Microsoft (and Apple?) have to make a serious attempt to embrace the open-source community or miss out on a lot of cool new software and developers. Times have changed somewhat; the mantra used to be "doesn't work on Linux", while nowadays we've got a lot more heterogeneous environment.

[lmm]

If those platforms are causing issues would it be worth downgrading them to a second-class "LTS releases only" state or some such?

[anongoogler2]

I could imagine implementing something like this as an alternate wire format and client library for protocol buffers. Can you outline the main reason you chose not to go this route? In particular, what aspects of the proto descriptor language don't fit Cap'n Proto? Or what aspects did you feel needed to be changed for some other reason?

[kentonv]

To be clear, by "alternate wire format and client library for protobuf" you mean "reuse the protobuf IDL but change everything else", right?

There are a few major reasons I didn't go that route:

* The .proto language has a lot of weird quirks that I don't like. Some of the quirks are specific to the protobuf encoding (e.g. int32 vs. sint32 vs. fixed32 being different types), while other quirks have no particular rationale behind them. I didn't want that baggage.

* The .proto language does not treat interfaces (aka services) as a first-class type. That is, you cannot define a field whose type is an RPC interface type -- a reference to a remote object. The ability to do this is a critical part of Cap'n Proto's interface design.

* It's highly unlikely that the protobuf team would be interested in accepting changes to the language which were not actually supported by protobuf. This means that if I shared the language, I would have my hands tied when it comes to new features -- or I'd also have to implement Protobuf equivalents to make them happy.

[anongoogler2]

Yeah that's what I meant. To me re-using the IDL would be a big plus because then all data can always be converted from one format to the other.

The reasons you gave make sense, although I'm not super familiar with the RPC aspect of protocol buffers so that's new to me.

[kentonv]

For cases that don't use features specific to one format or the other, it should be easy to write a tool that automatically converts between schema formats, FWIW.

[lobster_johnson]

I'd like to see Ruby bindings (not just serialization). We are using Go, Node.js and Ruby in production, and we have been looking to move from plain HTTP to gRPC. If Cap'n Proto is better, we might use it, but not without Ruby bindings.

[kentonv]

I don't think anyone is currently working on Ruby bindings, but if you're interested, feel free to take it on!

(Note that Sandstorm is focused on making Cap'n Proto work well for Sandstorm. We welcome contributions, but we generally don't have resources available ourselves to work on third-party feature requests unrelated to Sandstorm. That is, unless you want to pay us a bunch of money, in which case, feel free to contact me. ;) )

[bascule]

Actually there was a Capnproto project for Ruby GSoC project this year!

https://github.com/nemoNoboru/capnp-ruby

https://rubygems.org/gems/capn_proto-rpc

[nemoNoboru]

Yep, that's my project. Please use it and contact me if somenthing doesn't work!

[kentonv]

Hmm, should this be linked on the "other languages" page? I don't remember you telling me it was ready -- sorry if I missed something!

[lobster_johnson]

Awesome, thank you!

[lux]

I recently ran into flatbuffers https://google.github.io/flatbuffers), which claims the same no-serialization step. Curious how the two compare?

[bchallenor]

I recently evaluated the two and went with FlatBuffers, largely because its Java support appeared to be more mature.

The FlatBuffers encoding is based on vtables and is relatively straightforward (the runtime library is tiny). This also means it's inefficient for small messages, but in my testing its vtable deduplication worked great for my use case (~100k messages of the same type per memory-mapped file), in that the vtable overhead tends quickly to zero.

Cap'n Proto has a more complex encoding that is probably more efficient in terms of wire size, and particularly for small/standalone messages, but the runtime is larger as a result.

[adrianratnapala]

Does flatbuffers have an RPC mechanism? Or is it purely a serialisation tool?

Cap'n Proto is both. I.e. you have the serialisation, which can be used on its own, but you can also define interfaces with methods that can pass those serialised structs as parameters, and which return asynchronous promises.

[DenisM]

How about first-class map / dictionary support? That is, const or log time retrieval of key-value pairs.

Right now I can only see a way to create lists, so access will be in linear time.

[kentonv]

The "lists" are actually arrays, so access by index is constant-time. You can build a hashtable on top of that. I'd like to add built-in support for maps at some point, but it's one of a very large number of wishlist items...

[DenisM]

The simplest implementation would be a sorted list with binary search for access. Segments create problems though.

I thought of the hashtable, but that means I have to keep the hash-table in RAM. By contrast, the entire capnproto structure itself could be memory mapped, thus not having any RAM constraints.

I'm thinking larger datasets here, large enough where overhead of e.g. JSON becomes a problem with RAM (on mobile devices), but not yet large enough to have to use SQLite.

I might be overthinking it, and could move to SQLite straight away. Just trying to keep my stuff as simple as possible. Conceptually simple, that is. Capnproto is conceptually simple: a tree dumped to disk, allowing on-demand memory mapped access to requested parts.

[kentonv]

The capnp structure itself could be a hashtable, and doesn't need to be (entirely) loaded into RAM.

I'm saying you create a capnp list, and then you store elements into the list at position according to their hash -- i.e. how you'd build a hashtable, but the capnp list itself is the backing array.

You would have to do this at write time, of course, and make sure the hashing is consistent between runs.

[DenisM]

Do'h. Of course.

I'd still prefer the framework do it for me. It seems quite involved.

Thanks for taking time to answer my questions!

[easytiger]

From reading the examples, it looks like one can only serialise to a file descriptor. What if i wan to serialise to a byte array? Am I perhaps missing something obvious?

[kentonv]

You can certainly serialize to a byte array. The example uses a file descriptor but in fact the code defines an abstract stream type which you can implement any way you want, and you can also obtain pointers to the message's underlying storage in order to extract the bytes directly (avoiding a copy).

[easytiger]

Great thanks. I'll give it a go again for a current project I'm kicking off.

[easytiger]

i've had a look in kj/io.h

I see the classes: ArrayOutputStream/VectorOutputStream, i assumed these are the ones you are discussing.

[_0w8t]

I am curious, what is your take on Thrift? Is it really an improvement on protocol buffers?

[kentonv]

Thrift to me is a Protobuf clone. Very similar design. Early on it was not very well optimized compared to protobuf, but I imagine they've fixed that by now. The big advantage they had was that they included an RPC system in their first release -- although it was a FIFO RPC system which struck me as an odd design choice (I think they may have fixed this more recently?). But now GRPC exists, so I don't think there's much reason to choose Thrift over Protobuf/GRPC.

However, I am obviously very biased. :)

[zump]

Why did you leave Google?

[kentonv]

Well, I could write a lot about this, but...

In general, because I felt there was too much resistance to me pursuing ideas that I wanted to pursue. Google has become increasingly top-down, whereas it was fairly bottom-up when I started in 2005. That's not necessarily a bad thing, but I felt that I personally would be happier running a startup where I could call the shots.

FWIW, if you just want to write code, be comfortable, and make a crapton of money, and you don't care if you're implementing someone else's ideas, I highly recommend working for Google. That's not meant to be sarcastic or disparaging -- I totally respect that approach and there are days when I wish that were me. But if you have ideas of your own and you won't be happy unless you see them implemented... it probably won't happen at Google.

(To be fair, some people at Google would surely argue that my problem is that my ideas are crazy and bad, and I don't have any firm evidence -- yet -- that they are wrong.)

[zump]

Thanks for the candid reply!

[DonHopkins]

> and you don't care if you're implementing someone else's ideas and working for an advertising company

FTFY

[DonHopkins]

I'd be fascinated if someone who downvoted this would also be brave enough to explain why they're in denial that google is an advertising company.

It may not be that your ideas are crazy and bad, but rather that your sane good ideas simply aren't appropriate in the context of advertising.

Enough with the Stockholm syndrome. There are pleanty of perfectly great ideas that Google would never pay their employees to work on, however explicable or inexplicable the reason might be.

>A blow for mobile advertising: The next version of Safari will let users block ads on iPhones and iPads [1] : [...] An Apple realist might argue that its great rival Google makes more than 90 percent of its revenue from online advertising — a growing share of that on mobile, and a large share of that on iPhone. Indeed, Google alone makes about half of all global mobile advertising revenue. So anything that cuts back on mobile advertising revenue is primarily hurting its rival. (Google has been less friendly to adblockers than its “open” positioning would suggest.)

[1] http://www.niemanlab.org/2015/06/a-blow-for-mobile-advertisi...

[kentonv]

I didn't downvote you (in fact, I can't, since you were replying to me), but:

Your post was snide. Snide remarks are commonly downvoted on HN, regardless of their accuracy.

Regarding your follow-up, you're taking a very simplistic view of Google that doesn't really capture reality. Google obviously builds lots of technology that doesn't directly affect advertising revenue. Most decision-makers at Google are not asking "how does this affect advertising revenue" for every decision. You'd know that if you ever worked there.

[dragonwriter]

> I'd be fascinated if someone who downvoted this would also be brave enough to explain why they're in denial that google is an advertising company.

Several reasons, but most critically:

The phrase "advertising company" generally refers to a firm in the business of creating advertising. Google's main business is an online media company selling advertising space (both in its own advertising-supported services and alongside online media provided by others.)

[DonHopkins]

Is what the article says not true that "Google makes more than 90 percent of its revenue from online advertising" and "Google alone makes about half of all global mobile advertising revenue"?

That sounds like a pretty major advertising company to me.

Can you name a bigger advertising company than Google?

If they're actually a technology company driven by the demands of their users and striving for technical excellence, then do you expect Google to support built-in ad-blocking in Chrome and Android like Apple already supports it in Safari and iOS any time soon?

[zump]

Personally, I find it very difficult to motivate myself to apply for Google. Because of your reasons, and also becoming the 50,000th engineer. Will take almost ~2 yrs to work on anything interesting (Let's face it, most engineers aren't the world famous engineer in the Google Brain team).

[adrianratnapala]

... some people at Google would surely argue that my problem is that my ideas are crazy and bad ...

To what extent are such disagreements purely technical (e.g. byte-oriented wire encodings vs. roughly-like-memory) and to what extent are they "political".

By "political" I am talking about your claims about decentralisation and diversity, as if you are selling Sandstorm as the IBM-PC for the cloud millennium: a neutral platform any old software developer. That is very different from Google's way of doing doing business.

[kentonv]

I didn't actually propose massive decentralization while at Google, nor did I propose zero-copy encoding. So, no, I don't think I was proposing anything that threatened Google's business model.

[oneloop]

What a great pitch, that website is. I don't even understand what it is, but I'm stocked.

[crossroads091]

"Stoked". Sorry for the nitpick.

[oneloop]

I meant I'm in stock.