[sgarland]

> You're always going to need someone in your team who understands the tech and how to make best use of it.

…

> And that knowledge sits rather close to knowledge of how the system works, so given you'll need that knowledge anyway, may as well cultivate it instead.

This has been my argument forever, and it’s always met with disagreement, because entirely too many people have no desire to learn their tooling. They just want an API that they can push data into, and get it back out. What happens inside is irrelevant.

It’s extremely sad to me.

[redserk]

Or hold off on the academic-style pretentiousness and come back down to the real world.

At some point, we have to decide that there's a lot of knowledge expectations depending on your stack, especially as parts of your application grows.

Say you're a Python-based webapp running with Postgres, Kafka, and Elasticsearch. Your stack requires pretty decent knowledge of:

1. Postgres

2. Kafka

3. Elasticache

4. Linux (and a lot more than what many developers I've encountered seem to have)

5. Kubernetes, because it is 2024

6. Whatever frameworks you're doing with your webapp + ensuring you're keeping up with security best practices

7. + the soup involved with exposing your webapp to customers

Being able to handle any of these 6 at scale require different skillsets. It's unreasonable to expect anyone to be an expert at all of this -- in a real, tried-and-true environment -- especially with deadlines and SLAs involved.

[sgarland]

Counterpoint: stop the sprawl. Use boring technology.

Until you’re at quite a high scale, you probably don’t actually need Kafka. There are plenty of much lighter ways to do pub/sub, including Postgres itself.

Similarly, if your RDBMS schema is properly defined and your queries are well-written, you probably also don’t need Redis / EC.

Re: K8s, if you do need it, I’m not sure why people think that it’s so much easier to run EKS than your own cluster. The only thing you get to skip is the control plane; everything else is still your responsibility. Same with Postgres – you still are wholly responsible for its schema/table maintenance and optimization on major DBaaS.

In any case, nowhere did I say one person should be an expert at all of this.

[EdwardDiego]

> Until you’re at quite a high scale, you probably don’t actually need Kafka.

As someone who accidentally specialised in Kafka... ...bingo.

So many companies using it who don't need the sheer scale it offers, and get to pay the complexity cost anyway, with no benefits.