[zaptheimpaler]

Looks awesome, the work on performance especially is much appreciated :D I would love to be able to get flame graphs on my spark cluster per node too - any ideas on how exactly FB does this?

[threeseed]

Looks they are using the same library from Presto. You would just add the Java agent line to your Spark config i.e. spark.executor.extraJavaOptions.

https://github.com/prestodb/presto/issues/4004

Although you have to distribute the library across to all of the data nodes (if using YARN) or Spark nodes (if in standalone).

[coredog64]

Thanks! I was going to ask where I might find it since Google results were a little thin on the ground.

[peterstjohn]

I knocked up a quick Ansible script this morning. It is _very_ rough right now and is more like 'here's what you could do' (and you have to compile perf-map-agent beforehand), but I got it working on a small cluster: https://github.com/falloutdurham/spark-flame