[ehsantn]

I think staying in Python is best to ensure the team's productivity. There are a lot of tools that help accelerate and scale Pandas code while trying to be as compatible as possible. You can look at "Pandas on Spark", Snowpark (if on Snowflake), Bodo, Dask, and Modin for example.

Disclaimer: I work on Bodo and wanted to share it in case others find it useful. https://github.com/bodo-ai/Bodo