That's how safety critical devices are already built, so yes. We have standardized probabilities of failure (e.g. SIL [0]) from the unexpected, because mitigating 100% of risk is somewhere between impractical and impossible.
From a quick reading of the wiki, the associated methodology seems rather limited:
"System complexity, particularly in software systems, making SIL estimation difficult to impossible"
"The requirements of these schemes can be met either by establishing a rigorous development process, or by establishing that the device has sufficient operating history to argue that it has been proven in use."
You could prove that normal code satisfies some specs, but you can't do that with neural nets unless the number of possible inputs is tiny. So, the only way to establish that the black box neural net meets some SIL target is through "sufficient operating history".
To clarify, I wasn't offering SIL up as an example of how we should validate ML systems, but instead to demonstrate that "software 1.0" systems are already designed the way GP is questioning. Best practices for applying integrity level concepts to ML is still a topic of active debate right now.
"System complexity, particularly in software systems, making SIL estimation difficult to impossible"
"The requirements of these schemes can be met either by establishing a rigorous development process, or by establishing that the device has sufficient operating history to argue that it has been proven in use."
You could prove that normal code satisfies some specs, but you can't do that with neural nets unless the number of possible inputs is tiny. So, the only way to establish that the black box neural net meets some SIL target is through "sufficient operating history".