[obeliskora]

There was neat post https://news.ycombinator.com/item?id=27016630 a while ago about about using sqlite on static pages with large datasets that wouldn't have to be loaded entirely. Does duckdb do something similar with arrow/parquet files or its own format?

[ankoh]

Yes we do! DuckDB-Wasm can read files using HTTP range requests very similar to the sql.js-httpvfs from phiresky.

The blog post contains a few examples how this can be used, for example, to partially query Parquet files over the network.

E.g. just visit shell.duckdb.org and enter:

select * from 'https://shell.duckdb.org/data/tpch/0_01/parquet/orders.parqu...' limit 10;

[1egg0myegg0]

NIIIICE! Data twitter was pretty excited about that cool SQLite trick - now you can turn it up a notch!

[tomrod]

Is data twitter == #datatwitter, like Econ Twitter is #econtwitter?

If so, I have another cool community to follow!

[obeliskora]

It would be really cool to load duckdb files too. sql.js-httpvfs seems convenient because it works on everything in database so you don't have to create indexes, or setup keys and constraints in the client.

[ankoh]

I agree! DuckDB-Wasm can already open DuckDB database files in the browser the very same way.

[obeliskora]

That's really neat! Can you control the cache too?

[ankoh]

DuckDB-Wasm uses a traditional buffer manager and evicts pages using a combination of FIFO + LRU (to distinguish sequential scans from hot pages like the Parquet metadata).