[camgunz]

Oh good, another CBOR thread. Disclaimer: I wrote and maintain a MessagePack implementation. I've also bird dogged this for a while, HN search me.

Mostly, I just want to offer a gentle critique of this book's comparison with MessagePack [0].

> Encoding Details: CBOR supports indefinite-length arrays and maps (beneficial for streaming when total size is unknown), while MessagePack typically requires fixed collection counts.

This refers to CBOR's indefinite length types, but awkwardly, streaming is a protocol level feature, not a data format level feature. As a result, there's many better options, ranging from "use HTTP" to "simply send more than 1 message". Crucially, CBOR provides no facility for re-syncing a stream in the event of an error, whether that's network or simply a bad encoding. "More features" is not necessarily better.

> Standardization: CBOR is a formal IETF standard (RFC 8949) developed through consensus, whereas MessagePack uses a community-maintained specification. Many view CBOR as a more rigorous standard inspired by MessagePack.

Well, CBOR is MessagePack. Carsten Bormann forked MessagePack, changed some of the tag values, wrote a standard around it, and submitted it to the IETF against the wishes of MessagePack's creators.

> Extensibility: CBOR employs a standardized semantic tag system with an IANA registry for extended types (dates, URIs, bignums). MessagePack uses a simpler but less structured ext type where applications define tag meanings.

Warning: I have a big rant about the tag registry.

The facilities are the same (well, the tag is 8 bytes instead of 1 byte, but w/e); it's TLV all the way down (Bormann ripped this also). Bormann's contribution is the registry, which is bonkers [1]. There's... dozens of extensions there? Hundreds? No CBOR implementation supports anywhere near all this stuff. "Universal Geographical Area Description (GAD) description of velocity"? "ur:request, Transaction Request identifier"?

The registry isn't useful. Here are the possible scenarios:

If something is in high demand and has good support across platforms, then it's a no-brainer to reserve a tag. MP does this with timestamps.

If something is in high demand, but doesn't have good support across platforms, then you're putting extra burden on those platforms. Ex: it's not great if my tiny microcontroller now has to support bignums or 128-bit UUIDs. Maybe you do that, or you make them optional, but that leads us to...

If something isn't in high demand or can't easily be supported across platforms, but you want support for it anyway, there's no need to tell anyone else you're using that thing. You can just use it. That's MP's ext types.

CBOR seems to imagine that there's a hypothetical general purpose decoder out there that you can point to any CBOR API, but there isn't and there never will be. Nothing will support both "Used to mark pointers in PSA Crypto API IPC implementation" and "PlatformV_HAS_PROPERTY" (I just cannot get over this stuff). There is no world where you tell the IETF about your tags, define an API with them, and someone completely independently builds a decoder for them. It will always be a person who cares about your specific tags, in which case, why not just agree on the ext types ahead of time? A COSE decoder doesn't need also need to decode a "RAINS Message".

> Performance and Size: Comparisons vary by implementation and data. CBOR prioritizes small codec size (for constrained devices) alongside message compactness, while MessagePack focuses primarily on message size and speed.

I can't say I fully understand what this means, but CBOR and MP are equivalent here, because CBOR is MP.

> Conceptual Simplicity: MessagePack's shorter specification appears simpler, but CBOR's unification of types under its major type/additional info system and tag mechanism offers conceptual clarity.

Even if there's some subjectivity around "conceptual simplicity/clarity", again CBOR and MP are equivalent here because they're functionally the same format.

---

I have some notes about the blurb above too:

> MessagePack delivers greater efficiency than JSON

I think it's probably true that the fastest JSON encoders/decoders are faster than the fastest MP encoders/decoders. Not that JSON performance has a higher ceiling, but it's got gazillions of engineering hours poured into it, and rightly so. JSON is also usually compressed, so space benefits only matter at the perimeters. I'm not saying there's no case for MP/CBOR/etc., just that the efficienty/etc. gap is a lot smaller than one would predict.

> However, MessagePack sacrifices human-readability

This, of course, applies to CBOR as well.

> ext mechanism provides less structure than CBOR's IANA-registered tags

Again the mechanism is the same, only the registry is different.

[0]: https://cborbook.com/introduction/cbor_vs_the_other_guys.htm...

[1]: https://www.iana.org/assignments/cbor-tags/cbor-tags.xhtml

[aidenn0]

> ...awkwardly, streaming is a protocol level feature, not a data format level feature.

Indeed. I recall that tnetstrings were intentionally made non-streamable to discourage people from trying to do so: "If you need to send 1000 DVDs, don't try to encode them in 1 tnetstring payload, instead send them as a sequence of tnetstrings as payload chunks with checks and headers like most other protocols"

> Warning: I have a big rant about the tag registry. > ...

I completely agree with your rant w.r.t. automated decoding. However, a global tag registry can still potentially be useful in that, given CBOR encoded data with a tag that my decoder doesn't support, it may be easier for a human to infer the intended meaning. Some types may be very obvious, others less so.

e.g. Standardized MIME types are useful even if no application supports every one of them.

[camgunz]

> However, a global tag registry can still potentially be useful in that, given CBOR encoded data with a tag that my decoder doesn't support, it may be easier for a human to infer the intended meaning.

Yeah if MP is conservative and CBOR is progressive, I'm slightly less conservative than MP: I'd support UUIDs and bignums. But again, they'd have to be very optional, like in the "we're only reserving these tags, not in any way mandating support" sense.

[zzo38computer]

> This refers to CBOR's indefinite length types, but awkwardly, streaming is a protocol level feature, not a data format level feature.

BER also has indefinite length as well as definite length, but the way that it is doing, is not very good (DER only uses definite length). I think it is more helpful to use a different format when streaming with indefinite length is require, so I made up DSER (and SDSER) which is working as follows:

- The type, which is encoded same as DER.

- If it is constructed, all items it contains come next (the length is omitted).

- If it is primitive, zero or more segments, each of which starts with one byte in range 0x01 to 0xFF telling how many bytes of data are in that segment. (The value is then just the concatenation of all segments together.)

- For both primitive and constructed, one byte with value 0x00 is the termination code.

> Bormann's contribution is the registry, which is bonkers [1]. There's... dozens of extensions there? Hundreds? No CBOR implementation supports anywhere near all this stuff.

It should not need to support all of that stuff; you will only use the ones that are relevant for your program. (There is also the similar kind of complaint with ASN.1, and the similar response that I had made.)

> If something is in high demand, but doesn't have good support across platforms, then you're putting extra burden on those platforms. Ex: it's not great if my tiny microcontroller now has to support bignums or 128-bit UUIDs.

Although it is a valid concern, you would use data which does not have numbers bigger than you need to be, so it can avoid such a problem. You can treat UUIDs like octet strings, although if you only need small numbers then you should use the small numbers types instead, anyways.

> If something isn't in high demand or can't easily be supported across platforms, but you want support for it anyway, there's no need to tell anyone else you're using that thing.

Sometimes it is useful to tell someone else that you are using that thing, although often it is unnecessary, like you said.

[elcritch]

> Well, CBOR is MessagePack. Carsten Bormann forked MessagePack

Sure, that’s sort of true but missing context. Bormann (and others) wanted to add things such as separate string and byte sequence types. The MessagePack creator refused for years. Fair enough it’s his format. But it frustrated the community dealing with string vs bytes issues. It also highlights a core philosophical difference of a mostly closed spec vs an extensible first one.

> changed some of the tag values, wrote a standard around it, and submitted it to the IETF against the wishes of MessagePack's creators.

That’s just incorrect and a childish way to view it in my opinion.

The core philosophy and mental models are different in key aspects.

MessagePack is designed as a small self mostly closed format. It uses a simple TLV format with a couple hundred possible user extensions and some clever optimizations. The MP “spec” focuses on this.

CBOR re-envisioned the core idea of MessagePack from the ground up as an extensible major/minor tag system. It’s debatable how much CBOR is a fork of MPack vs a new format with similarities.

The resulting binary output is pretty similar with similar benefits but the core theoretical models are pretty different. The IETF standard bares little to no resemblance to the MessagePack specification.

> The facilities are the same (well, the tag is 8 bytes instead of 1 byte, but w/e); it's TLV all the way down (Bormann ripped this also).

The whole point of CBOR is that the tags go from 1-8 bytes. The parser designs end up fairly different due to the different tag formats. I’ve written and ported parsers for both.

It’s not like the MessagePack creator invented TLV formats either. He just created an efficient and elegant one that’s pretty general. No one says he ripped off “TLV”.

You can’t just take a message pack parser and turn it into a CBOR one by changing some values. I’ve tried and it turns out poorly and doesn't support much of CBOR.

> This refers to CBOR's indefinite length types, but awkwardly, streaming is a protocol level feature, not a data format level feature.

The indefinite length format is very useful for embedded space. I’ve hit limits with MessagePack before on embedded projects because you need to know the length of an array upfront. I wished I’d had CBOR instead.

This can also be useful for data processing applications. For example streaming the conversion of a large XML file into a more concise CBOR format would be much more memory efficient. For large scale that’s pretty handy.

> > However, MessagePack sacrifices human-readability > This, of course, applies to CBOR as well.

For the binary format yes. However the CBOR specification defines an official human readable text format for debugging and documentation purposes. It also defines a schema system like json-schema but for CBOR.

Turns out “just some specs” can actually be pretty valuable.

[camgunz]

I am really glad you replied.

> Sure, that’s sort of true but missing context. Bormann (and others) wanted to add things such as separate string and byte sequence types. The MessagePack creator refused for years. Fair enough it’s his format. But it frustrated the community dealing with string vs bytes issues.

msgpack-ruby added string support less than a month after cbor-ruby's first commit [0] [1]. The spec was updated over two months before [2]. Awful lot of work if this were really just about strings.

> It also highlights a core philosophical difference of a mostly closed spec vs an extensible first one.

MP has been always been extensible, via ext types.

> That’s just incorrect

I am entirely correct [3].

> MessagePack is designed as a small self mostly closed format.

Isn't it a lot of effort to get an IETF standard changed? Isn't that the benefit of a standard? You keep saying "mostly closed" like it's bad. Data format standards in particular really shouldn't change: who knows how many zettagottabytes there are stored in previous versions?

> It’s debatable how much CBOR is a fork of MPack vs a new format with similarities.

cbor-ruby is literally a fork of msgpack-ruby. The initial commit [0] contains headers like:

    /\*
     \* CBOR for Ruby
     \*
     \* Copyright (C) 2013 Carsten Bormann
     \*
     \*    Licensed under the Apache License, Version 2.0 (the "License").
     \*
     \* Based on:
     \*\*\*\*\**/
    /*
     \* MessagePack for Ruby
     \*
     \* Copyright (C) 2008-2013 Sadayuki Furuhashi

> The resulting binary output is pretty similar with similar benefits

This is the whole game isn't it? The binary output is pretty similar? These are binary output formats!

> but the core theoretical models are pretty different.

I think you're giving a little too much credence to the "theoretical model". It's not more elegant to do what cbor-ruby does [4] vs. what MP does [5] (this is my lib). I literally just use the tag value, or for fixed values I OR them together. The format is designed for you to do this. What's more elegant than a simple, predefined value?

> The whole point of CBOR is that the tags go from 1-8 bytes.

The tags themselves are only 1 byte, until you get to extension types.

> The parser designs end up fairly different due to the different tag formats.

The creator of CBOR disagrees: cbor-ruby was a fork of msgpack-ruby with the tag values changed.

> No one says he ripped off “TLV”.

Don't conflate the general approach with literally forking an existing project.

> You can’t just take a message pack parser and turn it into a CBOR one by changing some values.

This is a strawman. My claim has been about the origins of CBOR, not how one can transmute an MP codec to a CBOR codec.

> I’ve hit limits with MessagePack before on embedded projects because you need to know the length of an array upfront.

When everything's fine, sure this works. If there are any problems whatsoever, you're totally screwed. Any protocol that supports streaming handles this kind of thing. CBOR doesn't. That's bad!

> For example streaming the conversion of a large XML file into a more concise CBOR format would be much more memory efficient.

It's probably faster to feed it through zstd. Also I think you underestimate how involved it'd be to round-trip a rich XML document to/from CBOR/MP.

> However the CBOR specification defines an official human readable text format for debugging and documentation purposes.

Where? Are you talking about Diagnostic Notation [6]? Hmm:

"Note that this truly is a diagnostic format; it is not meant to be parsed. Therefore, no formal definition (as in ABNF) is given in this document. (Implementers looking for a text-based format for representing CBOR data items in configuration files may also want to consider YAML [YAML].)"

YAML!? Anyway, it literally doesn't define it.

[0]: https://github.com/msgpack/msgpack-ruby/commit/60e846aaaa638...

[1]: https://github.com/cabo/cbor-ruby/commit/5aebd764c3a92d40592...

[2]: https://github.com/msgpack/msgpack/commit/5dde8c4fd0010e1435...

[3]: https://github.com/msgpack/msgpack/issues/129#issuecomment-1...

[4]: https://github.com/cabo/cbor-ruby/blob/5aebd764c3a92d4059236...

[5]: https://github.com/camgunz/cmp/blob/master/cmp.c#L30

[6]: https://www.rfc-editor.org/rfc/rfc8949.html#name-diagnostic-...