That's a weird prediction to make, considering that PaLM-E does exactly that: https://palm-e.github.io/