[leemoonsoo]

Yes, if you already have k8s cluster that you don't manage, and you can use for your Spark, I would like to say you have Spark serverless environment.

IMO, where actual spark executor runs (either on lambda or k8s, or somewhere else) doesn't much matter to say it is serverless or not, while user can access the cluster, without managing them.

In our case, we build Spark Serverless service through

  - Fully managed Kubernetes cluster
  - Isolation between Spark instances (network, storage access, etc)
  - A dedicated Kubernetes node (VM) allocation for a executor, to provide container level security and better performance
  - Secure tunneling between remote Driver and executors for interactive mode (spark client deploy mode)
  - Various optimizations