[jstephan]

Thanks, great question !

Dynamic allocation is only enabled on our Spark 3.0 image (from the 3.0-preview branch, since the official 3.0 isn't released yet). It works by tracking which executors are storing active shuffle files. These executors will not be removed when downscaling. More info here: https://issues.apache.org/jira/browse/SPARK-27963

It's not perfect, but there are more improvements for dynamic allocation being worked on (remote shuffle service for Kubernetes).