Good discussion of why you can't stream VR experiences [0]. Also, streaming VR kills privacy.
Positional tracking with a smartphone's built in camera is already possible[1] thanks to SLAM[2]. Let me say that again, Google Cardboard and GearVR can (technologically) do positional tracking, right now.
The reason they don't is probably because you have to spend time scanning your room. But that's a really poor reason, I agree with OP, we should expect to see this happen very soon.