[moffkalast]

So, it has to rely on exact situational step by step commands? I'm wondering how one could conceivably deploy this in a useful way. Usually you'd need to mark areas on the map and then the robot knows where to go, A* is trivial around obstacles once you have that and a lidar. And lidars are an order of magnitude cheaper than something that can run an 8B VLA.

One could maybe autogenerate these text planning commands, but it would require a map and the robot's current location, so it doesn't really solve that, unless it can find a specific thing completely on its own. How much of a planning horizon does it have?

[mike_hearn]

You probably don't need a geometric map. Just have someone wander around with a mobile app and feed the video into a more powerful model once, asking it to produce descriptions of the different areas of the office or building and how they connect. Now you have a "text adventure game" map you can use it with a small LLM to produce instructions for the robot to follow, assuming it knows where it currently is.

The advantage over traditional approaches is presumably flexibility. LIDAR isn't going to solve an instruction like "find the man with the pink shirt".

[moffkalast]

Yeah the benefits of a model that can understand text well enough is certainly that one could in theory just describe the layout. Just let it know what connects to what, what's on each floor, etc. Might work if it can figure out where it is based on contextual info, but probably not super well yet I imagine. VLAs are known to have very little text context, like sentences worth at most, maybe this one's different.