[marcinzm]

>You don't find Kubernetes needlessly complicated? Do you actually need Docker or Kubernetes?

They make life easier in my experience especially if you need something more than a simple self-enclosed application. For what I do we need Apache Spark, Airflow (which in turn uses RabbitMQ and Celery if you want to scale out), possibly JupyterHub (which also needs some way to spin up child servers), and then the application code itself.

I've wired some of it up myself in the past and it's a massive pain in the backside with lots of edge cases (Spark used to have a shitty massive python script for cloud deployments which always was missing something). Deploying it on Kubernetes with Docker on the other hand is rather simple. I basically trade the complexity of Kubernetes/Docker for the complexity of custom deployment/scaling/management code. In my experience, the code quality of the former is going to be much much higher than the code that an internal DevOps team outputs.

[tedmiston]

I've spent timing wiring up a similar setup with Airflow, and Celery on Kube, Kafka too, and running Spark clusters and echo the same sentiment. I also find Kubernetes easier/simpler compared to DC/OS, perhaps because it's a bit more structured and opinionated.