Hacker News new | ask | show | jobs
by rdeboo 1677 days ago
I think the comparison was Snowflake vs Databricks SQL. Databricks SQL is a PaaS service just like Snowflake. Also, it uses their Photon engine, which is a proprietary engine written in C++. It is not Spark.
1 comments

I'm aware that Databricks is a PaaS service, but what Databricks runs under the hood is Spark (with a few proprietary extensions). So your jobs/queries do require some tuning just like with OS Spark.

Spark has had SQL engines (SparkSQL/Hive on Spark) for a long time. Photon is just a new, faster one. Photon tasks also run on Spark executors only, so it's not independent of Spark[1]. Also, while it's proprietary now, I wouldn't be surprised if Databricks open-sources it in the future, like they did with Delta Lake.

1. https://databricks.com/blog/2021/06/17/announcing-photon-pub...