[Towaway69]

You probably could but what would you see? A bunch of weights, connections between layers and more numbers.

You don't see any meaningful, understandable code. For example, if prompt begins with draw me a picture then jump to layer X.

I'm no expert but I can imagine that to be the problem when one attempts to debug an Algorithmic Intelligence black box.