| Snowflake's `Snowpark` product that they recently announced, which is to bring Spark-like APIs to Snowflake. Having a DS background, I love what SQL-orchestration tool dbt (and peers) have enabled: data consumers to rapidly create our own safe data pipelines. There's easily a 10x productivity improvement for most of my transformation pipelines vs. when I write them in Python or PySpark. But batch ML and SQL are not that friendly (even BigQuery ML is too limiting). I end up butchering dbt's value (simplicity and iteration speed), splitting the DAG into pieces and orchestrating them with Airflow so that I can wedge in other non-dbt parts (like feature engineering, inference, logging, detecting stale models, ...). This isn't what the future looks like. I've tried switching to Databricks, but do not see this as the path forward for unioning the warehouse + batch ML. Hopefully Snowpark is a step forward :) ------------------- Separately, https://materialize.com/ is something I'm paying attention to! Being able to implement all of my SQL-based pipelines as materialized views would be immensely valuable. They recently raised capital and they could become huge. |