[kstenerud]

It's because it was easy to use, and so people adopted it long before realizing just how many features they needed that it doesn't provide (for example, lack of types). "Simplicity" is a double-edged sword because you can't actually eliminate natural complexity; you can only move it around into (hopefully) more convenient / less chaotic forms.

I've been developing https://concise-encoding.org over the past two years with this in mind.

[willtim]

A huge improvement over JSON, but you still implement Tony Hoare's billion dollar mistake! There also appear to be no schemas / static types or sum types (the dual to records)? A schema would allow to you to overload the string literal, rather than require all those prefixes.

A data-exchange format is essentially developing a mini-language. Even if we prohibit abstraction, we still want rich static types and the ability to define new types (algebraic types, nominal types). Taking inspiration from programming language theory, will hopefully help avoid a result like Google Protobufs, which is awkward, ad-hoc and non-compositional.

[kstenerud]

Null is in there because it exists in the real world (unfortunately). There's no putting the genie back in the bottle (although there's an admonition against using it in the spec). Pretending it doesn't exist would cause more trouble in the real world than letting it be.

The string-type prefixing is there so that you don't actually need a schema for 80% of use cases (schemaless data is usually enough). A schema can be added later if desired, and I've been thinking of what that might look like, but it's at a different level than the encoding format and can be developed independently.

The primary purpose of this format is to serve the 80% use case (general purpose, ad-hoc, hierarchical, schemaless data with fundamental type support - or something that easily fits within this paradigm). For the remaining more complicated 20%, there are custom types and at some point in the future, schemas.

[willtim]

> Pretending it doesn't exist would cause more trouble in the real world than letting it be

I don't agree at all with this point. RDBMS types for example are non-nullable by default. Protobufs, XML and many other exchange formats also have optional types but not null added to every type. If you want better interoperability with programming languages, I think it would be better to go after e.g. IEE754 support.

[kstenerud]

The default nullability of RDBMS types is implementation-specific. Protobufs requires a lot of ugly workarounds for languages such as Java to handle the language's nullable types. You can't eliminate the complexity; only move it, and protobufs moved it in a way that punishes languages with nullable types. XML at least works via omission, but that doesn't handle the case of explicitly signalling "no data", which is a valid signal in many languages.

Concise Encoding does support IEEE754.

[willtim]

> You can't eliminate the complexity; only move it,

Yes and by using null-in-every type for your messaging, now you've pushed the problem out of your app and published it to the world.

[kstenerud]

Precisely.