Philosophers call this ability to predict another agent's behavior "theory of mind". Presumably it would be impossible to have a theory of mind - a theory about what another agent will do - without first being aware of your own.
To me it seems like you can have a state of mind, and use that state to predict the state of mind of others without being _aware_ of your state of mind at the same time - that is, without being conscious.
In principle, yes. In practice, it would be much harder to rig a system that can be aware of others' states of mind without being aware of its own. Nature never does things the hard way when the easy way works. Consciousness is the easy way.
Computers can make predictions about conscious agents and even revise their predictions based on past data, a counterpoint to the idea that agents cannot be predicted without theory of mind.
Unless I'm showing my ignorance here, which is quite likely.
I think it’s the other way around: we evolved the ability to build a theory of mind (to improve our social standing), and later applied that theory-of-mind to ourselves, greatly amplifying & reifying consciousness.
I fail to see why that would be true. I think there are different definitions of consciousness here that are being mixed up.
> Without consciousness you experience a single unified reality: your reality.
To me the word "experience" is what consciousness seems to be about. In that case without consciousness you can "step outside" and everything, but you would have no "internal experience" of any sort. i.e. be a "philosophical zombie" [1].