[telescopeh]

Considering what type of sensors that would need to be are already on these robots, would it not be possible to teleoperate within VR? So the motion and control can be more fulid but there is a lag before it is applied to real robot. Like how mosh[0] works.

[0] https://mosh.org

[jononor]

It i possible to have a digital twin which simulates the real physics of the robot and gives instant feedback based on that. It might help a bit to make the teleoperator work within the capabilities of the robot, be more in sync. But is peobsbly not realistic to simulate the real world physics faithfully, in such a way that operators can use it as actual feedback. Especially in sensitive scenarios like grabbing a glass, there are tipping points (sometimes literal), where a few millimeters and 100 milliseconds is the difference between close call and full smash.

Thinking about it now, if one would deliberately add much more latency (a few seconds), it might be possible to use real-world simulation as aid. At least for operations which can be decomposed into sequences of transitions between stable/safe states. Say moving dishes from dishwasher to cupboard. Picking up is critical, but holding in hand is (presumably) safe, placing in cupboard critical, once placed it is safe. Then one could let teleoperator do the entire critical move virtually, act it out in simulator only. See what the outcome is. If high risk of failure, deny operation. If good chance of success (per simulation) can allow to execute in the real world. More autonomous operation will need ability to simulate actions, project alternative approaches into the future, and a world model strong enough that can also plan and execute based on it. So there are potential synergies in a full-teleop, to hybrid teleop to autonomy transition. Note, this approach would also assume the relevant environment to be static. So it would not help handle the pet or toddler...