[eternityforest]

Why not just use msgpack? The advantage of JSON is that support is already built in to everything and you don't have to think about it.

If you start having to actually make an effort to fuss with it, then why not consider other formats?

This does have nice backwards compatibility with existing JSON stuff though, and sticking to standards is cool. But msgpack is also pretty nice.

[benatkin]

This seems to be geared towards using a heavily adopted format.

Some would want to move to binary, but it's hard to find an ideal universal binary format.

msgpack doesn't support a binary chunk bigger than 4gb, which is unfortunate. Also the JavaScript library doesn't handle Map vs plain object.

In JSON you could have a 10GB Base64 blob, such as a video, in a string, no problem (from the format side, with a library YMMV).

For one that supports up 64 bit lengths, check out CBOR: https://cbor.io/ With libraries maybe it could be the ideal universal binary format (universal in the same sense JSON is - I've heard it called that). https://www.infoworld.com/article/3222851/what-is-json-a-bet...

[klabb3]

> msgpack doesn't support a binary chunk bigger than 4gb, which is unfortunate.

I don’t care what behemoths people store in the formats they use but at the point you exceed “message size” the universality of any format is given up on. (Unless your format is designed to act as a database, like say a SQLite file.)

> In JSON you could have a 10GB Base64 blob, such as a video, in a string, no problem

Almost every stdlib json parser would choke on that, for good reason. Once you start adding partiality to a format, you get into tradeoffs with no obvious answers. Streaming giant arrays of objects? Scanning for keys in a map? How to validate duplicate keys without reading the whole file? Heck, just validating generally is now a deferred operation. Point is, it opens up a can of worms, where people argue endlessly about which use-cases are important, and meanwhile interop goes down the drain.

By all means, the stateful streaming / scanning space is both interesting and underserved. God knows we need something. Go build one, perhaps json can be used internally even. But cramming it all inside of json (or any message format) and expecting others to play along is a recipe for (another) fragmentation shitshow, imo.

[serial_dev]

Example: you work on the mobile team, the backend team is large and focuses on serving the web app, they send huge JSON payloads that the mobile app only partially need, and asking the backend team to now also serve msgpack is out of the question as things together with the backend and web teams were proven to be a PITA.

In this scenario, writing a new, or bundling someone else's json library can significantly improve things.

[lpapez]

Sounds like a dysfunctional organization to be honest. Why couldn't you agree on the contents of the mobile API? If the backend team is slow, why can't the mobile team implement a proxy towards the backend to serve only the data they need?

[serial_dev]

I totally agree, it was a dysfunctional org, but sometimes fixing the org is extremely hard and you want to make progress today focusing on what you can do to improve the user's experience, instead of focusing on what others should change and do.

[OnlyMortal]

I worked at TomTom on the Home application. There was a similar problem with map updates where the payload was XML.

The devices only needed a sub-range of the XML so I used an XML parser to ignore everything until I got the tag needed then read until the end tag arrived.

This avoided a DOM and the huge amount of memory needed to hold that. It was also significantly faster.

[secondcoming]

That’s an issue with your teams rather than the message format.

Even sending data that the mobile app doesn’t need raises flags.