[avogar]

Using ClickHouse you can also process local files in one line using clickhouse-local command tool. And it will look a lot easier:

clickhouse local -q "SELECT passenger_count, COUNT(*), AVG(total_amount) FROM file(taxi.csv, 'CSVWithNames') GROUP BY passenger_count"

And ClickHouse supports a lot of different file formats both for import and export (you can see all of them here https://clickhouse.com/docs/en/interfaces/formats/).

There is an example of using clickhouse-local with taxi dataset mentioned in the post: https://colab.research.google.com/drive/1tiOUCjTnwUIFRxovpRX...

[eatonphil]

Clickhouse-local is incredible. It does the best of any similar tool I've benchmarked. But the reason I took it out of the linked benchmarks in OP's post is because it's 2+GB. That's a massive binary. It's the whole server. I'm not sure you want to be distributing this all over the place in general. It's just not in the same category IMO. Disclaimer: I build another tool that does similar things.

[zX41ZdbW]

It is shipped with debug info and symbol tables just in case.

Without debug info, it will be 350 MB, and compressed can fit in 50 MB: https://github.com/ClickHouse/ClickHouse/issues/29378

It is definitely a worth improvement.

[eatonphil]

Aha, thanks for the clarification and link. I'll be following that issue.

[qoega]

It is that big only with debug symbols. If you take packaged version of ClickHouse it will be much smaller. Or just strip large binary manually if you already have it.