[tartoran]

So you’re hoping for a blackbox uninspectable by humans? That to me sounds like a nightmare, a nightmare worse than all the cruft and stupid rules humanity accrued over time. Let’s hope the future tech is inspectable and understandable by humans.

[idiotsecant]

I think if we assume that AGI will be a thing the odds of future tech remaining inspectable by humans is pretty unlikely. Would you build a car so that your dog can maintain it?

[tartoran]

Fully understandable end to end by any normal human and inspectable enough for human governance are different things. In any sane world, AGI would be built inside a human institutional environment: laws, audits, liability, safety engineering, access controls, operational constraints, etc. We do not build planes so passengers can reconstruct the turbine from scratch, but we still require them to be inspectable by the people responsible for certifying/repairing them. The right standard is not whether an average person can rebuild or fully undestand the whole machine, but whether human institutions can reliably inspect, verify and govern it. If they can’t, then the technology is not mature enough to trust.

[lelanthran]

> So you’re hoping for a blackbox uninspectable by humans?

We already have that. He's hoping that the blackbox gets smart enough to understand itself.