[sandGorgon]

in case you are using PySpark - a good framework to move to is Dask. https://docs.dask.org/en/latest/

it is also natively integrated with K8s - https://github.com/dask/dask-kubernetes - and Yarn https://yarn.dask.org/en/latest/