[tjah1087]

SIMA author here - SIMA is betting on simulations and games. We use real-world open-ended language, but games/sim obviously have simplified physics/graphics.

In terms of alternative strategies, Google DeepMind also has an amazing robotics team with lots of fantastic work for real-world robotics - including multi-robot generalists, showing positive effects when co-training one agent or model on multiple environments/bodies. Their prior work was very inspirational to us in SIMA! https://deepmind.google/discover/blog/scaling-up-learning-ac...

[hnhn34]

Sorry if this is a stupid question but even after reading OP’s post, I’m having trouble understanding the limitations of this.

What would happen if, for example, the system were trained on simple 3D puzzle games, with natural language instructions such as “solve this puzzle” with a hint like “X or Y strategy might work”? I see that menu navigation is part of the training set. Can this thing learn to “read” or does it just learn the results of menu actions? If it can learn to play No Man’s Sky.. can it learn to play Zelda?