[brunohaid]

Good point and was mostly re Typesense (can't edit the comment anymore).

But given that duckdb handles "take this n GB parquet file/shard from a random location, load it into memory and be ready in < 1 sec" very well I'd argue it's quite easy to build something that scales horizontally.

We use it for both the importer pipeline that processes the 2B row / 200GB compressed GBIF.org parquet dataset and queries like https://www.meso.cloud/plants/pinophyta/cupressales/pinopsid... and the sheer amount of functions[1] beyond simple stuff like "how close is a/b to x/y" or is "n within area x" is just a joy to work with.

[1] https://duckdb.org/docs/stable/core_extensions/spatial/funct...

[steve_adams_86]

Duckdb is so absurdly portable you can solve all kinds of performance and scaling concerns this way. It's really a wonderful, fun piece of software.