I think the way GPS apps have resolved this is using your bearing, comparing it to lanes on the road, and moving the route to match where you are actually moving. This might need some finer adjustment based on where your camera is pointing and your bearing.