[usrbinbash]

> but also who is to say that somewhere in those billions of parameters there isn't something like a model of goals and emotional state?

No one, but as is well established, absence of proof of nonexistence isn't an argument for existence. https://en.wikipedia.org/wiki/Russell's_teapot

[m4nu3l]

Correct, but the problem is how you prove that for humans is by using the output and inferring that. You can apply the same criteria to ML models. If you don't, you need some other criteria to rule out that assumption for ML models.

[usrbinbash]

For humans I can simply refer to my own internal state and look at how I arrive by conclusions.

I am of course aware that this is essentially a form of Ipse dixit, but I will do it anway in this case, because I am saying it as a human, about humans, and to other humans, and so the audience can just try it for themselves.