|
|
|
|
|
|
by ibains
2168 days ago
|
|
|
Very valid points:
1) Agree that Snowflake is far easier to use than Spark.
2) Agree that DBT is a great tool. ETL workflows normally processing 10s of TBs and workflows with large and complex business logic is the context. With Spark code, you can break down your code into smaller pieces, see data flow across them, write unit tests, and have the entire thing still execute as a single SQL query. Don't large SQL scripts become really gnarly for complex stuff - nothing short of magical incantations. I can't see data flow from a subquery for debugging without changing code. Prophecy as a company is focused on making Spark significantly easier to use! |
|
|