[throwawaylala1]

I've worked at 3 organizations that adopted k8s. This all happened over quite a long period so I saw what k8s adoption looked like at varying stages of k8s development.

k8s gets shit for one very simple reason: it forces teams to deal with issues that they're not necessarily ready to deal with just yet. I've seen it happen over and over and over.

Small startups are able to get away with not using auto-scaling, having bad secret management, not running minimally sized containers, etc, etc. Are they going to solve these problems eventually? Yes, if they grow and don't die. But until they grow they simply can't afford dedicating a lot of effort setting up Hashicorp Vault with proper secret rotation (just as an example).

One of the companies I worked at was a 10 person startup that was trying to find product market fit at a break-neck pace. We had customers, big ones. It was a non-trivial piece of software too. We ran the whole thing on a very beefy EC2 instance. The DB and the application were on the same server. Provisioning an application server was all done with a single bash script. One time we had to re-provision this server and we had take everything down for like 3 hours. We did at 3am so none of our customers got disrupted.

Every single ounce of engineering time was devoted to product development. We ended up getting bought and everyone (all 10 of us; founders were generous and awesome) made a crap ton of money. I can tell you with absolute certainty that every major feature we shipped contributed to that acquisition. Had we dedicated any effort at all to proper infra it just wouldn't have happened. The cost of our crappy duck taped infra was that one night we needed to reprovision everything and maybe another 20 total hours of wasted time across all engineers in the company.

That's why the first line in the blog post specifically cites early stage startups.

[aahortwwy]

> it forces teams to deal with issues that they're not necessarily ready to deal with just yet.

Exactly this. Polyrepos, microservices, and their ilk all do the same thing. Bring forward problems you may one day need to solve and make solving them necessary for progress.

[throwawaylala1]

Yea. We had a monorepo. The whole thing was one big webapp.

Even our background workers ran in the web app. Oh yea, that's right. I can feel the cringing just writing that! So like if we had an expensive background operation, our web app would start up a thread and do the work. Some might be wondering: "What if you needed to restart it? Like, I dunno... when deploying code?". Answer: we didn't, lol. Our deploy script would hold all new jobs and wait until existing ones finished before restarting. Some background process were extremely time consuming (eg: 10+ hours) and those we wrote in a way that they could just resume if killed.

It's actually amazing how productive you can be with a single monolithic web app on a beefy cloud instance.

I was also surprised how quickly we were able to scale out to a proper multi-service architecture after being bought. It only took us 6 months. The big company that bought us plans ahead years. It was no problem at all.

Now don't get me wrong. I'm not advocating for sloppy engineering here. Our team was some of the smartest people I've ever worked with. It wasn't uncommon for me (a medium-experienced engineer) to bring a proposal to our technical founder that would require like 1 month of work. That freakin' genius would reframe the problem and architecture such that we'd spend 2 days, still achieve the same goal, and take on minimal technical debt. I learned A LOT.

I know I'm rambling, but honestly I don't know if that kind of hustle is even a thing anymore. Are tech startups still scrappy? Are they focusing on the core problems/solutions they're trying to prove out instead of layers of architecture? Do such problems even exist anymore? Back in the mid-2000s it felt like that's how everyone did it.

[xyzzy_plugh]

There's nothing wrong with the architecture you describe. In fact, I've even migrated/merged microservices into a monolith several times. What you describe about restarting, etc. is accurate, and you need to solve for that even in other architectures.

Conway's law is more important: "I need to deploy my team's change without waiting for your team's job to finish".

[abrichr]

Thanks for sharing! Which startup / what was the problem / solution?

[throwawaylala1]

I'd rather not say but it wasn't anything mind blowing. It was a B2B thing with a great UI in the early 2010s. You'd never have even heard of us. All of this predated the popularity of k8s. Docker wasn't really a thing. AWS was just growing and blowing everyone's minds.

[abrichr]

> We ended up getting bought and everyone (all 10 of us; founders were generous and awesome) made a crap ton of money.

Sounds mind blowing enough to me :)

[akvadrako]

Nothing about using k8s forces you to use microservices or auto scaling or good password management. These are just strawmen.

[throwawaylala1]

Sure. We could also not use a container registry. Or resource management. Or liveness/readiness probes. For the database we could use RDS and take the latency/$$$ overhead. For file storage set up a separate EBS volume or whatever. But... why?