[haberman]

I'm not super familiar with the Java implementation, but I believe it's pretty optimized and appears to do very well generally on that benchmark.

One unavoidable issue is that, unlike JSON, protobuf serializers have to do two passes over the message tree, because in protobuf binary format all submessages are prefixed by their length. The first pass just calculates lengths, while the second performs the actual serialization. This could potentially slow down serialization compared to JSON, especially for message trees with lots of nodes/depth.

[abecedarius]

So encode from back to front? Then when you reach the front you know the length.

[haberman]

Interesting idea, and could be interesting to experiment with.

There are a lot of practical challenges though -- to provide a useful API you'd have to reverse the string at the end, since socket APIs don't generally provide streaming "WriteReverse" functions, and for good reason, because it would force them to buffer arbitrary amounts of data.

So we'd have to reverse the entire string at the end. The question is whether this would be cheaper than doing a second pass over the message tree. And also keep in mind that you would need the first pass to decode everything -- including UTF-8 data -- in reverse. But since the UTF-8 APIs for Java strings probably don't support this, you'd probably have to encode it, then reverse it to put it in the encoding buffer. That way when it gets reversed again at the end, it would be proper UTF-8.

At the end of the day, this probably wouldn't end up faster than what we do now. But can't say for sure without trying!

[abecedarius]

Reversing the output didn't come up when I did this sort of thing in C (for a different format): you can fill a buffer from the end, then just write the tail. But I guess Java does stick you with that? And I didn't have to deal with encoding to UTF-8 (ouch), or whatever other complications you might face, like streaming. Oh well! Hope the suggestion was stimulating anyway.