[espadrine]

Yes. The matrix is stored. In fact, there has been a number of visualization tools for neural nets to help track down which weights contribute to which results.

Additionally, the weights are constant in production, which makes analyzing production issues just a matter of logging the inputs and the outputs. Of course, for most convolutional nets, the inputs can get quite large (for example, a camera feed).