This is practically exactly what Protobuffers are. Except that they actually are defined clearly enough for multiple services written in multiple languages can work with them.
Definitely not, protobuf's strange wire format becomes apparent if you ever look at the hexdump of one or the profiler output of your favourite protobuffer-decoding C/C++ application.
They're actually kind of performance heavy for no benefit.
I have once looked at a benchmark that compared protobuffer, message pack, json and a variety of other serialization formats. In terms of reducing bytes per message gzipped json was ahead of all of them at the cost of increased CPU time for gzip. Protobuffer did pretty poorly, the only benefit was decreased CPU usage. I'm sure you could use some other compression algorithm like LZMA to get both good compression and good performance for JSON messages.
> In terms of reducing bytes per message gzipped json was ahead of all of them
Try gzipping the protobuf. Binary encoding and compression are different things which can be stacked. Gzipped protobuf should be smaller and faster than gzipped json in basically all cases.
I use LZ4 (with "best" compression) for packet captures and replay with great results.
I get about a 37% compression ratio with extremely fast decoding, like 10 million packets per second off an SSD.
It was better than snappy, gzip, and bz2 for the trade-off of compression time, decompression time and file size.
As for protobuf: flatbuffers, capn proto, HDF5, and plain C structs all deliver much, much faster decoding time. It's really not the best answer for any serialization at this point but it's still inexplicably popular.
Sure, but anything you're trying to transport between languages which don't even agree on endianess will end up like this.
Dumping a struct on a wire is just a wishful dream that turns into a nightmare as soon as you need to send that to a service written in another language or running on another architecture.
Don't get me wrong - there's plenty of insanity in protobufs. But trying to cover the same use-case will not create a simple protocol.
Cap’n’proto isn’t well supported apart from C or Rust.
Python library is an absolute nightmare. Their tests used to catch Exception, and what they ended up testing was basically whether their test try to access nonexistant attributes.
The issue is that capnproto is relatively more complex, and as such is harder to implement well.
They're actually kind of performance heavy for no benefit.