[mccanne]

This is a very real problem being addressed here and I am intrigued by all the great comments in this thread.

In the Zed project, we've been thinking about and iterating on a better data model for serialization for a few years, and have concluded that schemas kind of get in the way (e.g., the way Parquet, Avro, and JSON Schema define a schema then have a set of values that adhere to the schema). In Zed, a modern and fine-grained type system allows for a structure that is a superset of both the JSON and the relational models, where a schema is simply a special case of the type system (i.e., a named record type).

If you're interested, you can check out the Zed formats here... https://github.com/brimdata/zed/tree/main/docs/formats

[mccanne]

Also, if any of you find problems with the Zed spec(s), we'd love to hear about them. "Now" would be a good time to make changes / fix flaws.

[petre]

I'd like to see more examples and probably data serialized as zed.

[mccanne]

There are a few examples in the ZSON spec...

https://github.com/brimdata/zed/blob/main/docs/formats/zson....

And you can easily see whatever data you'd like formatted as ZSON using the "zq" CLI tool, but I just made this gist (with some data from the brimdata/zed-sample-data report) so you can have a quick look (the bstring stuff is a little noisy and an artifact of the data source being Zeek)... https://gist.github.com/mccanne/94865d557ca3de8abfd3eb09e8ac...