[lmeyerov]

Any sense of comparison to Dremio, which helped steward the Arrow ecosystem for doing this kind of thing?

(The idea is great fwiw, I've been following them one-off for years, and we have to do elements of these things in how we build louie.ai and Graphistry for the GPU equivalent. Real pain point!)

[lukekim]

Dremio is awesome. We've followed the Dremio journey from one of Jacques' original talks a couple of years back. Dremio's idea of caching tiers and reflections is powerful for performance.

Spice takes it further and provides flexibility for materialization, giving you full control over where that materialization exists (same machine, same pod, same network, same cluster, same region, etc.), what engine/processing (OLTP - SQLite/PostgreSQL, OLAP - DuckDB/Arrow) it uses and what tier (in-memory, attached NVMe, etc.) to store it down to the dataset level.