[gopalv]

Hidden partitioning is the most interesting Iceberg feature, because most of the very large datasets are timeseries fact tables.

I don't remember seeing that in Delta Lake [1], which is probably because the industry standard benchmarks use date as a column (tpc-h) or join date as a dimension table (tpc-ds) and do not use timestamp ranges instead of dates.

[1] - https://github.com/delta-io/delta/issues/490

[fiddlerwoaroof]

Delta Lake now has Hilbert-curve based clustering which solves a lot of the downsides of hive partitioning

[gopalv]

> Hilbert-curve based clustering which solves a lot of the downsides of hive partitioning

Yes, that solved the 2-column high NDV partitioning issue - if you had your ip traffic sorted on destination or source, you need Z-curves, which are a little easier with bit twiddling for fixed types to do the same thing.

Hive would write a large number of small files when partitioned like that or you lose efficiencies when scanning on the non-partitioned column.

This does fix the high NDV issue, but in general Netflix wrote hidden partitioning in specifically to avoid sorting on high NDV columns and to reduce the sort complexity on writes (most daily writes won't need any partitioned inserts at all).

While clustering on timestamp will force a sort even if it is a single day.

[autodidacticon]

What is NDV partitioning?

[artwr]

NDV = Number of distinct values. Here partitioning on high cardinality columns, essentially.