What is it giving you over an ec2 auto scaling group or ECS/Fargate?
Both can scale as much as you like, your config can live as cdk/cloudformation/terraform code?
I'm not an expert on auto scaling groups, but we have used the Google Cloud equivalent for a time. The biggest issue for us was that deployment is not as easy. Can you update the software on the ec2 instance without turning it on, for example? With K8s, we can leave the deployment scaled to 0 and just patch the image of the deployment to perform a release while the workers are all shut down. Similarly, we don't have to write code to wait for the workers to finish their current job before we shut them down in order to be replaced by a newer version; this is all managed by K8s, and the configuration for it lives in Git.
As others have already pointed out, it is also important for us to remain independent from Google Cloud / AWS.
Apache Spark can use Kubernetes as a scheduler out the box. I don’t know if op is using Spark.
A lot of data tools are starting to target Kubernetes directly as a runtime so using them with GKE/EKS is a bit simpler as it’s officially supported, allows to run locally and on the cloud with no vendor lock in.
ECS in a scaling group works well if your app is stateless but as soon as you scale workers dynamically, do service discovery, orchestration, you end up building some of the features Kubernetes provides.
Also something like karpenter.sh is much nicer than ec2 autoscaling group.