[namaria]

This whole thing makes me think of that short story "The Machine Stops".

As we keep burrowing deeper and deeper into an overly complex system that allows people to get into parts of it without understanding the whole, we are edging closer to a situation where no one is left who can actually reason about the system and it starts to deteriorate beyond repair until it suddenly collapses.

[IanKerr]

We are so, so far beyond that point already. The complexity of the world economy is beyond any one mind to fully comprehend. The microcosm of building black-box LLMs that perform feats we don't understand is yet another instance of us building systems which may forever be beyond human understanding.

How is any human meant to understand a billion lines of code in a single codebase? How is any human meant to understand a world where there are potentially trillions of lines of code operating?

[namaria]

When your house is on fire and someone says "get out", certainly grabbing a jerrycan of gasoline and dousing yourself in fuel is worst than just getting out?