|
|
|
|
|
by ithkuil
744 days ago
|
|
Define "same benefits". Bloom filters allow you to prune the number of files you even have to look at, which matters a lot when there is a coat associated with scanning the files. Partitioning the data can be advantageous both for pruning (on its sort keys) or for parallel query fan-out (you independently scan and apply the predicate to the high cardinality column in each partition concurrently). In the use case that underpins the article, they want to minimize unnecessary access to parquet file data because it lives on a high latency storage system and the compute infrastructure is not meant to be scaled up to match the number of partitions. So they just want an index to help find the data in the high cardinality column |
|