[singhrac]

We use DuckDB similarly but productionize by writing pyarrow code. All the modern tools (DuckDB, pyarrow, polars) are fast enough if you store your data well (parquet), though we work with not quite “big data” most of the time.

It’s worth remembering that all the modern progress builds on top of years of work by Wes McKinney & co (many, many contributors).

[wenc]

Yes Wes McKinney was involved in both Pandas and Parquet and Arrow.

[7thaccount]

I remember reading something awhile back that when building pandas he was getting a lot of inspiration from things like APL and I assume Kdb+.