[l_dopa]

Is knowing, for example, that there's a race triggered by a new optimization in compiler version X that happens under conditions Y and Z really a qualitatively different type of understanding than "the weights are such that the ANN computed this incorrect output for these inputs?"

In both cases we can only fix a particular instance of an error and effectively have no way to show that similar errors don't exist, or even that we didn't introduce new bugs with the fix.

[Klockan]

The difference is that all human written systems was once understood by a human. ANN on the other hand creates unlabeled spaghetti which gets impossible to reason about if you scale that up to megabytes. You could try to fix a bug by changing one of the values, but that would ruin the ANN since all the values were chosen to be optimal. Thus the only way to make it better would be to change lots of unlabeled values at the same time and that is something no human can do. The only plausible way to fix a bug in an ANN is to incorporate said bug in the training set.

[l_dopa]

> The only plausible way to fix a bug in an ANN is to incorporate said bug in the training set

My point was not something about manipulating weights directly vs. augmenting the training set. The analogy is roughly, find a misclassification : add particular input to training set :: find a bug : add a patch/test case. I'm playing devil's advocate, of course, and there are obvious, important differences.

> The difference is that all human written systems was once understood by a human

But that's not one of them! In practice, we create software systems by getting some rough intuition about an interface, wiring together some components, then checking the results empirically. Basic components that everything else is built on, C compilers for instance, are based on ambiguous, even contradictory specifications.

[jamescostian]

> find a misclassification : add particular input to training set

Unfortunately, adding that particular input to the training set may mess up the ANN when given a different input (i.e. you fix one bug but create one or more new bugs, like a game of whackamole but where people die)

[jimbokun]

"the weights are such that the ANN computed this incorrect output for these inputs?"

Can we actually do this, for complex ANNs?

[espadrine]

Yes. The matrix is stored. In fact, there has been a number of visualization tools for neural nets to help track down which weights contribute to which results.

Additionally, the weights are constant in production, which makes analyzing production issues just a matter of logging the inputs and the outputs. Of course, for most convolutional nets, the inputs can get quite large (for example, a camera feed).