[iliabara]

Hey, Ilia here :)

Not sure if the question is about the simulator Caladan, or about our autonomy core.

For Caladan, we don't support playback of data currently. Do you have a use-case need for this?

For the autonomy core, the localization, costmap generation, and path planner are all probabilistic to a more or less degree, so we can't guarantee determinism to any highly accurate degree. However, if the same data is played back over and over, the vehicle would react in basically the same way (perhaps not navigating to the exact same spot, but within a small tolerance). Same question here; what is your need/use case to have guaranteed determinism? Is it a safety question? (We can have a much longer chat there)

[captaindiego]

Thanks, I was talking more about the autonomy core side of things.

So to narrow it down a bit more, my question is a bit tied to safety, but also to general development. Given a set of logged inputs from a machine that had a field issue X, am I able to reliably reproduce (hopefully deterministically) what happened on that machine? (and therefore what went wrong)

[iliabara]

Ah ya, debugging in real environments is a very difficult question, particularly in robotics where the system is complex, and not always with the best internet connectivity.

The approach we are taking to this is two fold: 1) Robust self monitoring and metrics, ensuring we measuring system performance (we are using Prometheus to allow us to scale to (hopefully) millions of devices. 2) Ensuring we tie those metrics to a rolling buffer of the last 90 seconds (or so) of data. When an issue occurs, we automatically save to disk, and allow remote operators to pull the raw sensor data as needed.

With the raw data, we can (and do!) play this back through our stack and step through the behavior tree and control outputs, comparing them to what actually happened on device. The system is certainly deterministic enough to allow this sort of debugging and testing!

We will have a set of articles on this approach in the future!