A while back during a particularly rough patch where everything was going wrong, I started thinking, "man, I really hope I'm being stupid and doing it wrong..." (because then I can stop doing that!)
And wouldn't you know it, I keep getting my wish :)
I believe it's a great deal worse than that. All the metacognitive insight we do have may just be confabulation and we are fooled into believing that we have it because the process for conjuring it is good at finding a plausible answer.
When you read about and observe the split-brain patient experiments the appropriate response is abject horror at the implications.
It is a characteristic of neural nets that they do not have insight into their own functioning.
It is arguably a characteristic of any intelligent system, that at least some part of it must be opaque to itself, but the previous sentence is more defensible than a generalized claim.
If you don't understand what that means, tell me from your own metacognitive insight what parts of your brain are being used to read this. Not because of learned knowledge about what parts of the brain do what, through your own insight in your own functioning. You can't, because you don't have any.
This isn't just that human rationalize a lot. This is below that. This is that even if you notice yourself rationalizing, which is something you can train yourself to do, you have no access to the underlying computations/processes of the rationalization itself, or the process of noticing you are rationalizing.
There is arguably still a sense that we experience in which we humans could reasonably say "No, I'm pretty sure I used addition-with-carry to answer you", so that is perhaps not the easiest example to think about the experience of. But there will always be some question of "how did you do that" to which you can give no answer because the answer is in the firing of the neural net itself and you, who is in one way or another the product of that firing, do not have access to that. How did you quickly catch that ball that someone unexpectedly threw at you? You just did, as far your neural net is concerned.
(Also, while I've expressed this in terms of your conscious experience, this doesn't have anything to do with "consciousness". Neural nets in general do not get this feedback and do not and can not have arbitrary metacognition about their own functioning. This is an artifact of my writing text to address conscious beings.)
Yep. An effect cannot in itself reason about its cause. Some effects can suggest causes though. For example you tend to have memory of an algorithm you just executed, when the usage was at least somewhat conscious or intended, which can lead you to be able to guess which algorithm you used (and potentially even correctly)
As trivial as that example is, it boggles my mind just how large the scale gets of things about me I do not fully understand or cannot explain. It feels different than losing track of what I just did, because a memory of it seemingly never existed to lose in the first place. For example, as much as I can try to reason about executive dysfunction, I cannot seem to understand the real actual equation that results in me being willing or not to do something. It just feels like my own brain disagrees with me, and that's so frustrating, and I've been trying to rationalize it for years but in the end I just do not know, and likely cannot know.
It's not even trivial to identify what it is exactly I'm not aware of. There's just some pattern I don't like, and the factors that influence it are a mystery. I've discovered some things over the years that seem to correlate with it, but nothing that truly explains or remedies it.
I am not incapable of metacognition, but I still have observed (and keep observing) cases where I seem to lack real metacognitive insight into something.
And wouldn't you know it, I keep getting my wish :)