[nevi-me]

My experience teaching some graduates in a BI shop. SQL is more common, and tools that support SQL tend to be used better.

I've "taught" them how to use Spark, but being a team of varying prior experience, the Scala API meant them learning Scala, the Python one was a bit better, but they did much better with the SQL DSL.

Regarding your concern re maintainability: UDF's tend to be the problem, I'm also curious to know re their support issues, and also: can anyone write their own UDF (the code requires registering a .jar), or is there a team that helps business users in that regard?

[haohui]

For UDF we support both -- users can write their own UDFs in their queries. We provide a number of well-tested UDFs to them as well.