[querulous]

i see a lot of "spark is dead" talk here. what replaces it for transform inbetween something like kafka and redshift/bigquery?

[llbeansandrice]

I agree with both sides here. Spark was the in thing for a while so a lot of places are using it but probably don't need it and could have been better off running various SQL scripts to do some transformations. I worked on a project exactly like this where we should have used SQL scripts instead of Spark.

But I also think that a lot of enterprise pipelines went all in on spark and so now moving to something else (SQL scripts, Snowflake, etc.) just isn't worth it. So Spark is dead, long live Spark.