[smokel]

Simply add images and video, and these estimates start to sound like the "640 KB should be enough for everyone".

After that, make the robots explore and interact with the world by themselves, to fetch even more data.

In all seriousness, adding image and interaction data will probably be enormously useful, even for generating text.

[netcan]

Like both will be done. Idk what the roi is on adding video data to the text models, but it's presumably lower than text.

There are just a lot of avenues to try at this point.

[llSourcell]

no its not lower than text, its higher ROI than text for understanding the physics of the world, which is exactly what videos are better at than text when it comes to training data

[AstroBen]

Does that transfer, though? I'm not sure we can expect its ability to approximate physics in video form would transfer to any other mode (text, code, problem solving etc)

[ricopags]

depends on the hyperparams but one of the biggest benefits of a latent space is transfer between modalities