I'm curious: what do you do for developer environments? Do you have a need to spin up a partial subgraph of microservices, and have them talk to each other while developing against a slice of the full stack?
> Do you have a need to spin up a partial subgraph of microservices, and have them talk to each other while developing against a slice of the full stack?
Yeah we do. We use k3s, and use kustomize to scale down or disable services from prod. minio replaces S3, postgres runs nicely in a container, environment variables for service discovery. Add in a liberal sprinkling of NodePort services to allow connections from web browsers or whatever.
Been using it for about three years now. Works great. Upgrading is easy, rebuilding is easy.
We even use it to run our one-box testing environment on an EC2 instance, rather than spinning up an actual EKS cluster. Also works great.
I love that we use the exact same tools to manage all environments -- yaml+kustomize for manifests, kubectl+k9s for ops -- so we get really used to using them.
Can’t speak for everyone but I have worked in this environment. It can work fine if you allocate a sub slice of CPU time (.1 CPU for example) and small amounts of (overcommitted) memory, and explicitly avoid using it for things that are more easily managed by cloud provider sub accounts and managed services. IE don’t force your devs to manage owncloud or a similar stand in for S3 - use something first party to stand in or S3 itself.
This doesn’t always work and the failure mode of committing to this can be doubling your hosting bill if it won’t run locally and densely packed small instances can’t handle your app.
I would recommend Tilt + kind clusters (via https://github.com/tilt-dev/ctlptl) - minimum headache setup by a large margin and runs well on linux and macs
I love tilt, but it feels like I'm doing something wrong when I write a whole new tilt deployment yaml.
I _think_ this space just hasn't matured enough to have settled on standard solutions. Between dev, CI, and staging/prod, it feels like you're defining variants of the same objects and topologies over and over. Some things try to take on dev+CI or CI+deploy (dev+deploy anyone?), but I haven't seen an answer for the whole thing that "feels right".
That’s YAML problem not Tilt. There have been attempts to throw more template yaml at the problem (kustomize, etc) I myself prefer Starlark-based DSL like this one: https://github.com/cruise-automation/isopod
We've got pretty much a significantly scaled down mirror of production as a developer enviroment. The k8s development cluster looks the same, has the same workloads, tools, services etc. The AWS infra is the same but also scaled down. Its easy to have your service running in both enviroments and rapidly iterate on that. Everyone is actually pretty happy with that and we've never been let down by Kubernetes itself in at least 4 years now.
It's worked great for us. Every developer runs a dev cluster on their own machines. Services like s3 are transparently replaced with mock versions. We have two builds that can be run, which really just determines which set of helm charts to deploy: the full stack or a lightweight one with just the bare necessities.
Nah, nothing that complex. The only AWS service we use that isn’t just their clone of an existing tool is S3, which we just use fakes3 for. Everything else is easily deployable in a cluster already, because it’s mostly standard systems like mysql, elasticsearch, redis, etc, that AWS has versions of, but that don’t require any special treatment. The ingresses, etc are obviously different, because it doesn’t rely on the aws loadbalancers in dev, but that’s all abstracted away and automatically handled anyways.
Yeah we do. We use k3s, and use kustomize to scale down or disable services from prod. minio replaces S3, postgres runs nicely in a container, environment variables for service discovery. Add in a liberal sprinkling of NodePort services to allow connections from web browsers or whatever.
Been using it for about three years now. Works great. Upgrading is easy, rebuilding is easy.
We even use it to run our one-box testing environment on an EC2 instance, rather than spinning up an actual EKS cluster. Also works great.
I love that we use the exact same tools to manage all environments -- yaml+kustomize for manifests, kubectl+k9s for ops -- so we get really used to using them.