Mapillary has been able to recover 3d structure in areas with high photo density:
http://blog.mapillary.com/update/2015/11/10/pointclouds.html
That's still a long way from merging the photos into a coherent view, but you quickly start to be able to 'look around'.