[jonnat]

So, will Databricks deprioritize Delta Lake in favor or Iceberg, or will they try to derail Iceberg development now they they got the team that originally built it at Netflix?

Edit: from the Tabular CEO announcement

  Databricks reached out to me and proposed a collaboration that could bring Iceberg and Delta closer together [...] I’m excited to have the opportunity to work with Databricks and the broader Delta community to build better table formats together.

So it seems they are going for the latter.

[indoordin0saur]

Doesn't it sound like they'll try and move the two formats closer together so that there isn't such a format war? IDK how it would benefit Databricks to ruin either format if they're now such huge stakeholders in them both.

Either way, I just want to know which format to pick. I've been chief data engineer at my current company for about a year and would like to be able to move off of plain parquet files in my lake but I'm not sure what table format to choose.

[waterlx]

Hi, in case you did not find the answer yet. In my hamble opinion: - choose Iceberg: If you have several computing/query engines other than Spark, like Presto, Flink. Iceberg has a great extraction and design for a engine-independent table format. But its learning cost is relative high - choose Delta: If you only have Spark and would like to be deeply binded with Databricks - choose Hudi: If you would like to use data lake out-of-the-box and it is quite easy to use. - If your data is updated frequently, like streaming, check https://paimon.apache.org/ if you would like to be deeply binded with Flink

[indoordin0saur]

Thank you! Sounds like iceberg is the best then. I'm very allergic to lock-in. Currently we're very Spark heavy and our query engine is AWS Redshift Serverless. The recent AWS Glue Catalog support for Iceberg seems to make this promising.

[ruipds]

I heard from a AWS worker that they consider Iceberg to be the future. A lot of their services will be glued together with it.

[TheCleric]

At the Databricks Summit keynote this morning they pitched it as a way of trying to standardize and bridge across the two more easily, with neither going away.

[datadrivenangel]

So Delta will eventually become better but not before Databricks gets a year or three of advantage over iceberg and open source delta.