[colinator]

This seems perfect to hook up to my 'LLMs can control robots over MCP' system. The idea is that LLMs are great at writing code, so let's lean in to that. I'll give it a try! I just got a bigger robot, we'll see how it does...

https://colinator.github.io/Ariel/post1.html

[Glemllksdf]

Really unfortunate that I forgot what YT video i saw about this just 2 weeks ago.

It was about Googles PaLM-E evolution and progress. It basically has two models one which controls the robot, the other is a llm and they are combined together in some attention layer.

[johntb86]

Must have been https://www.youtube.com/watch?v=2mrGMMmrVNE from Welch Labs.

[colinator]

That video is pretty good, thanks for finding it. I'm basically betting that an earlier, abandoned approach described in the video, "Code as Policy", will beat everything else. It requires no training data, and generalizes instantly to all robots.

[Glemllksdf]

Yes :-)

[fennecfoxy]

I would look into VLAs/VLMs if you haven't already.