I saw that dynamic allocation is enabled by default. I thought dynamic allocation does not work well on k8s if the executors need to be kept around for serving shuffle files. How does it work in your case?
Dynamic allocation is only enabled on our Spark 3.0 image (from the 3.0-preview branch, since the official 3.0 isn't released yet). It works by tracking which executors are storing active shuffle files. These executors will not be removed when downscaling. More info here: https://issues.apache.org/jira/browse/SPARK-27963
It's not perfect, but there are more improvements for dynamic allocation being worked on (remote shuffle service for Kubernetes).
Dynamic allocation is only enabled on our Spark 3.0 image (from the 3.0-preview branch, since the official 3.0 isn't released yet). It works by tracking which executors are storing active shuffle files. These executors will not be removed when downscaling. More info here: https://issues.apache.org/jira/browse/SPARK-27963
It's not perfect, but there are more improvements for dynamic allocation being worked on (remote shuffle service for Kubernetes).