Wouldn't it be better to use a simulated environment first. Maybe something using pybullet[1] and a script which maps real world to STL or OBJ files[2]
Today's virtual worlds are not accurate enought to allow to develop perception algorithms in simulation. In order to develop sensor fusion, you also need to simulate the output of all sensors including their specific characteristics. Apart from the model quality, there is another challenge: Simulation runtime, which is substantial (!) and - to my best knowledge - not even close to realtime.
If you want to develop driving algorithms that sit on top of the perception stack, then this becomes simpler. You can work on the object level (object being simulated cars, pedestrians, ...) and statistically model perception errors. This is a lot faster, which is e.g. important, if you want to run large-scale reinforcement learning to develop your driving strategy.
In any case, in the future, I would say that we will see a lot more simulation (don't forget, all major players build heavily on simulation - just look into the numbers on how many miles Waymo simulates every day) and potentially going down all the way to the sensing level, because it allows you to develop and especially debug along the whole sense - plan - act stack.
Also, today there exist also hybrid approaches. You take real recordings and abstract them into a simulatable format that you can then, e.g., use to variate and derive artificial scenarios for simulation. This can be used to analyse the influence of different situative paramters on the behavior of a function to pinpoint which parameter(s) caused certain troublesome behavior that have been observed in real drives.
How would something like Waymo's daily simulation training work? Is it just feeding it a larger set of random obstacles every day?
Wouldn't it quickly have negligible returns once it optimizes for the current simulation capabilities? Or are they constantly tweaking both the car model and the simulation data set?
Today's virtual worlds are not accurate enought to allow to develop perception algorithms in simulation. In order to develop sensor fusion, you also need to simulate the output of all sensors including their specific characteristics. Apart from the model quality, there is another challenge: Simulation runtime, which is substantial (!) and - to my best knowledge - not even close to realtime.
If you want to develop driving algorithms that sit on top of the perception stack, then this becomes simpler. You can work on the object level (object being simulated cars, pedestrians, ...) and statistically model perception errors. This is a lot faster, which is e.g. important, if you want to run large-scale reinforcement learning to develop your driving strategy.
In any case, in the future, I would say that we will see a lot more simulation (don't forget, all major players build heavily on simulation - just look into the numbers on how many miles Waymo simulates every day) and potentially going down all the way to the sensing level, because it allows you to develop and especially debug along the whole sense - plan - act stack.
Also, today there exist also hybrid approaches. You take real recordings and abstract them into a simulatable format that you can then, e.g., use to variate and derive artificial scenarios for simulation. This can be used to analyse the influence of different situative paramters on the behavior of a function to pinpoint which parameter(s) caused certain troublesome behavior that have been observed in real drives.