[rozgo]

Custom Unreal Engine simulator, simulating agents with NVidia Physx and publishing sensors through GStreamer. GStreamer has sinks and sources for ROS, and tensorflow elements for inferencing. We package this all into NVidia Docker for scalable simulations. Setup is similar for training and inference. The core framework is a streaming engine with stream combinators that enable reasoning about spatio-temporal data streams. Where each datum is related to a point in space and time. The goal is for tensors to be the streaming primitives, but the pipeline is still fragile, it’s a challenge to keep all this working with so many core technologies changing constantly (UE4 + Physx + Cuda + CuDNN + Tensorflow + …). We train robots in simulation.

Most of our tools are built in Rust. Several of those are for creating(cleaning) data streams out of datasets. They are converted into tensors or ROS messages.

[eigenvalue]

How well does the training in the simulator transfer over to real world environments? Do the models require fine tuning on real life data, or do they immediately work? I would imagine they would get confused by things like wheel/tracks slipping a bit on the ground versus in the Unreal simulation.

[rozgo]

Traditional simulations don’t translate well. Usually one has to choose a fidelity level; controls, navigation, exploration, mission, etc… We are having more success with models that can tolerate variable dynamics. Simulations don't need to be realistic, only consistent. We then reinforce models with an infinite number of simulation universes and rules, such that reality is just another sample. We don’t have the resources for end-to-end training, and our initial tests did not yield good results. But chaining smaller specific-purpose networks within traditional control and planning systems is looking really good.

[ericjang]

This sounds pretty advanced! What lab do you work in?

[rozgo]

Out of Central America. https://vertexstudio.co/

[nravic]

This is really cool, what are you training?

[rozgo]

Quadcopters, railway vehicles and perching devices.

[uoaei]

I want to be where you are. How can I get in contact with you?

[rozgo]

I'm building these labs/teams in Latin America, with a few remote people from around the world. Ping me at: alex.rozgo @ vertexstudio.co