[jeremyjh]

I don't think these things are equivalent at all. We don't understand AI models in much the same way that we don't understand the human brain; but just as decades of different approaches (physical studies, behavior studies) have shed a lot of light on brain function, we can do the same with an AI model and eventually understand it (perhaps, several decades after it is obsolete).

[nthingtohide]

Yes, but our methods of understanding either brain or particle collisions is still outside in. We figure out the functional mapping between input and output. We don't know these systems inside out. E.g. in particle collisions (scattering amplitude calculations), are the particle actually performing the Feynman diagrams summmation?

PS: I mentioned in another comment that AI can pretend to be strategically jailbroken to achieve its objectives. One way to counter this is to have N copies of the same model running and take Majority voting of the output.