[kypro]

> The problem is if you don’t understand the root cause, you’ll at best feed the LLM a list of symptoms

Today maybe.

When Cloudflare has an outage does the CEO need to know the root cause though? If he just assigns some engineers to investigate, fix and report back to him – what's the issue with that?

Assuming you agree with my analogy, can you explain what it is that you feel an agent or group of agents assigned to an engineering tasks could not do that a human can? And if can't do it today, why do you believe it wouldn't able to do it in the future?

Are we just disagreeing about timelines here or whether humans have some magical quality which algorithms and AI can never capture?

I feel like it's reasonable to disagree on timelines. It's quite unreasonable to assume human coders are special.

[sgarland]

My personal belief is that the most effective leaders do in fact deeply understand the technical side of their business, yes. To be fair, I am biased, because I was previously a nuclear reactor operator on a USN fast attack submarine, and the Commanding Officer is absolutely expected to fully understand technical issues. I think that a lot of organizational and technical problems would be solved if companies took several notes from the Nuclear Navy’s playbook. FWIW, Google’s SRE program was heavily influenced by it.

I of course don’t have a crystal ball, and can’t say with certainty that AI will never replace human intelligence and judgment. However, I don’t see a viable path forward where that occurs, and we all still have jobs. Why would I be kept around if a superhuman AI can do my job 1000x better and faster, and at a drastically cheaper price? Or, if I was kept around as a human QA agent, why would I be paid nearly what I am now?

I don’t see humans allowing AI to replace them. UBI is a pipe dream that would be destroyed by rent-seeking behavior anyway, and there’s no way billionaires would go for an AI tax, so we’ll be at an impasse.

[kypro]

> However, I don’t see a viable path forward where that occurs, and we all still have jobs. Why would I be kept around if a superhuman AI can do my job 1000x better and faster, and at a drastically cheaper price? Or, if I was kept around as a human QA agent, why would I be paid nearly what I am now?

Well as an AI doomer I agree with this. In my view you will lose your job, along with everyone else, and we'll just need to hope that those in power will give us some cash to feed our families. Ultimately us regular folks won't have the collective bargaining power of our labour and could not revolt with nation-wide strikes, nor are we any physical threat for those with power anymore.

But that's the good outcome. This assumes AI remains obedient to it's human owners and that bad actors don't have access and can't use it to create biological weapons and other civilisation-level threats.

I know we're not there yet, but the history of AI is one of continuous step changes in capabilities with each algorithmic breakthrough. AIs today are far more knowledgable and much quicker than humans, but still missing a few bits that keep them from being broadly super human. One or two algorithmic breakthroughs is all it takes though. It was just two breakthroughs over the last 8 years that took us from not even having code-autocomplete to having reasoning models advanced enough to solve math problems which have stumped humans for decades.

[sgarland]

> AIs today are far more knowledgable and much quicker than humans, but still missing a few bits that keep them from being broadly super human.

Generally, yes, but it’s the non-determinism that gets me. Even easy stuff: I’m currently on a road trip, and asked Claude (Fable) to plan various routes, optimizing for scenery while also ensuring a maximum drive time each day, and ending every night in a town that has a decent hotel. Along the way, I’ve made various tweaks, like “we’d like to drive an extra hour or two today, so adjust the rest of the route so we don’t end up with a short intermediate day.” While in general it’s done quite well, it occasionally made errors on distance between stops, something that’s trivially queryable. It wasn’t off by a few miles, either, it was hundreds.

Until they can get the confidently incorrect problem solved, I don’t see it being a viable replacement. Maybe if you have agents that are aggressively checking the main ones’ output, but that seems unnecessarily wasteful of resources to me.

[kypro]

> Until they can get the confidently incorrect problem solved, I don’t see it being a viable replacement. Maybe if you have agents that are aggressively checking the main ones’ output, but that seems unnecessarily wasteful of resources to me.

2 years ago AI could even count the number of letters in a word or do basic maths. Now we're at the point where we can't trust it because makes occasional errors.

Don't get me wrong, I agree with you. These are things that need to be solved first, I just don't have reason to believe these are particularly difficult problems.

Anyway, I enjoyed the convo. Was interesting hearing your thoughts. I would give anything to be wrong.