[xrjn]

It appears that someone has already done something like this [0]. In 2013 during 30C3 in Hamburg, someone bought in a bicycle which you used to bike through cities virtually using an Oculus rift [1]. Unfortunately I couldn't find any extra information about that project, but I do seem to remember that they had to cache everything locally because Google blocked them after 10000 API calls a day.

[0] https://github.com/troffmo5/OculusStreetView [1] https://instagram.com/p/ibU1IXFhCz/