The Kalman filter can be derived as the conditional mean/variance of a multivariate normal. You assume a linear state-space model and walk the equations forward. Those are the key limitations: linear state space; Gaussian innovations. You can derive it other ways, but that's the way I grok it.
You are correct about hidden states. A linear state-space model with omitted variables will suffer the same kinds of bias present in an OLS model with omitted variables [4].
Deriving the equations is a nice way to distract yourself from the apocalypse. [1],[2] should be enough of a toe hold if you are familiar with OLS. Ignore the control term u[n] in the Matlab documentation. Kalman's original paper [3] is also a really nice, although I didn't really get it until I had already approached it as a conditional moment problem.