They are essentially Apache Spark-as-a-service and have an API that allows you to submit a job on a cluster that you can configure to autoscale: https://community.cloud.databricks.com/doc/api/#jobs.JobsSer... https://community.cloud.databricks.com/doc/api/#jobs.Cluster...