Ask HN: Autonomous Cars Simulation

[coderunner]

There's a couple companies in the business of using a 3d simulation for training autonomous cars like Waymo, Cruise, Nvidia, and Applied Intuition. I don't quite understand their product though.

1. Are the trained object detectors in the simulation applied to real world data also or is only the part that makes decisions transferred to the real vehicle (e.g. it's safe to turn left here) while detectors trained on real world images of cars, people, etc. used?

2. Tangentially, I thought that in general detectors trained on computer generated images was not very applicable to real world images. eg training on a bunch of images of 3d modeled humans won't work well with testing on pictures of real humans. Is this not true?

[hacoo]

I work on full-3D simulations (using a game engine) for an autonomous car company. I can't speak for every AV company, but in my experience, simulators are used far more for testing than 'training'. The appeal of using a 3D game engine for simulation is that you can create inputs to the car's perception system. Without this ability, you're stuck either replaying recorded data, or spoofing out perception and only testing planning/controls and down. These two approaches are actually extremely powerful, so the vast majority of AV simulation testing is not done in full 3D.

There are some situations where 3D simulation is useful, though. First, it allows you to run your AV software in its entirety (i.e., not spoofing perception), making for a very complete integration test. A 3D sim can capture complex, interesting occlusions that other sims cannot. Another fairly common use case is experimenting with new sensor setups before they're added to the car.

As for training, it's mostly research at this point. I think there's promise in using synthetic data to supplement real-world data training data for perception systems.

There are a number of companies trying to market simulation 'platforms' to AV makers. I think there's the potential for one of these products to gain traction -- but it's a difficult sell. AVs are enormously complicated, a 3rd party product would need to both beat in-house sims and support a lot of very specific (and likely propriety) AV features.

[coderunner]

Thanks for answering! Does that mean the perception system that's trained on real world data does well on the synthetic data coming in during testing? Or are the perception systems' training data kept separate (synthetic data for training the perception system used when testing in the simulation and real world data for training the real world perception system)?

And yeah, I've come across quite a few companies working on simulation platforms in my reading (cognata, metamoto, applied intuition, etc.) which made me interested in what they're selling but actual details are quite sparse on their websites.

[Datenstrom]

Simulation to real training is an active area of research and I'm not aware of it being used anywhere in production for critical systems. I have not seen anything better than the "Learning Dexterity"[1] paper that was published last year.

[1]: https://openai.com/blog/learning-dexterity/

[coderunner]

That was my impression as well. Maybe the idea is get the models trained in a simulation to a satisfactory level and work on transfer of the model to real world data in parallel? There's an awful lot of money going into the first part if there's no good answer to the second part right now though.

[natch]

Taking just the images subset of this, but it should apply for other types of data as well:

It doesn’t have to be computer-generated images. It can also be computer—altered images (think n° rotation, blurring, cropping, etc.) which should work pretty well in part because real world images are sometimes rotated, blurred, cropped, etc.

[coderunner]

I just found this paper related to that actually

https://arxiv.org/abs/1901.07849

From what I can find out seems like all the companies are using unreal or unity for rendering the graphics though.