[pauldix]

Yeah, Parquet is awesome. One of the things we really want to do here is to push DataFusion (the Rust based SQL execution engine) to work on Parquet files, but to push down predicates and other things and operate on the data while it's compressed.

You pay such a high overhead marshalling that data into an Arrow RecordBatch. Best thing ever is to work with the Parquet file and not even decompress the chunks that you don't need. Of course, this assumes that you're writing summary statistics as part of the metadata, which we plan to do.

[nevi-me]

There's rudimentary statistics support, but I've found that it accounts for a lot of the write time (I wrote some tests last weekend, I can ping your team when I put them on GH).

Improving our stats writing could yield a lot of benefits. I'll open JIRAs for this in the next few days.