[zellyn]

Interesting. I wonder how different that is from RecordIO. Also, whether there'll be a Go implementation.

[Edit, after looking a bit.]

Pretty different. If I remember correctly, RecordIO is re-synchronizing, whereas Riegeli seems to break things up into 64KB chunks, splitting messages across chunks if necessary.

[Edit, after finding more information.]

Interesting… looks like Riegeli is intended to compress well, rather than just store sequentially. https://encode.ru/threads/2895-Riegeli-%E2%80%94-a-new-compr...

[throwaway84742]

IIRC (but memory has faded considerably), RecordIO also did support something to aid compression across records (rather than just offer per-record compression). There was some gnarly code in it to that effect where there could be a compressed subset of several records within the file. But I might be wrong.