[hitradostava]

Thanks :-)

I've not done much profiling on DuckDB and what the overhead is - i.e. after the data is parsed how much memory is used. Would be really interesting to push it to the limit - or to explore not loading the entire file in, but only reading the relevant parts, but again that probably requires a conversion first, e.g. to parquet or some other column based storage format.

[prasoonds]

that sounds like a fun project :) maybe I'll take a stab at it!