[apoverton]

I've thought about solving this problem with an ML approach like you all are taking but as you say never had the bandwidth because I was focusing on my "core missions". I'm no longer a heavy spark user but am very happy to see you all working on this!

It always seemed so inefficient to me to spend all this time hand tuning jobs only to have the data change and need to do the same thing again.

Good luck!

[jstephan]

Thanks for the wishes! Indeed it's rarely worth it to build an automated tuning tool: - Unless you operate at a massive scale (eg Dr Elephant + TuneIn projects, originally developed at LinkedIn) - Or you operate a big data platform yourself.

If you're curious about our ML approach, we gave a tech talk about it at last year's Spark Summit: https://databricks.com/session_eu19/how-to-automate-performa...