Hacker News new | ask | show | jobs
by dtech 93 days ago
It's not quite clear to me why you'd use this over something more established such as protobuf, thrift, flatbuffers, cap n proto etc.
2 comments
maxmcd 93 days ago
Those care about quickly sending compact messages over the network, but most of them do not create a sparse in-memory representation that you can read on the fly. Especially in javascript.

This lib keeps the compact representation at runtime and lets you read it without putting all the entities on the heap.

Cool!

link
IshKebab 93 days ago
Amazon Ion has some support for this - items are length-prefixed so you can skip over them easily.

It falls down if you have e.g. an array of 1 million small items, because you still need to skip over 999999 items to get to the last one. It looks like RX adds some support for indexes to improve that.

I was in this situation where we needed to sparsely read huge JSON files. In the end we just switched to SQLite which handles all that perfectly. I'd probably still use it over RX, even though there's a somewhat awkward impedance mismatch between SQL and structs.

link
creationix 93 days ago
I did seriously consider SQLite, but my existing datasets don't map easily to relational database tables. This is essentially no-sql for sqlite.
link
creationix 93 days ago
Exactly. Low heap allocations when reading values is one of the main driving factors in this design!
link
konart 93 days ago
What if you are reading from a service which already have an established API?

It's not like you can just tell them to move to protobuf.

link
SV_BubbleTime 93 days ago
What about CBOR that can retain JSON compatibility?

If you are working with an end you don’t control, this “newer better” format isn’t in your cards either.

link
creationix 92 days ago
How does CBOR retain JSON compatibility more than RX?

RX can represent any value JSON can represent. It doesn't even lose key order like some random-access formats do.

In fact, RX is closer to JSON than CBOR.

Take decimals as an example:

JSON numbers are arbitrary precision numbers written in decimal. This means it can technically represent any decimal number to full precision.

CBOR stores numbers as binary floats which are appriximations of decimal numbers. This is why they needed to add Decimal Fractions (Tag 4)

RX already stores as decimal base and decimal power of 10. So out of the box, it matches JSON

link