[martythemaniak]

Yeah, afaik the approach people take today is always some form of bi or tri level hierarchical control, with a slow LLM doing planning and sub task management and diffusion or VLA doing the motor control at higher frequencies. Major differences seem like where and how you draw the boundaries. For my project I'm personally trying to use ROS2 as a low level tool call (instead of diffusion), with an agent /LLM doing the main decisions.

Having said that, this scheme seems like it might just be a reaction to current hardware limitations. When I saw Talaas demonstrate a 8B model running on a custom chip at 17k Tok/sec, first thing I thought was "wow, you can just run an LLM in a control loop"