Did some light digging and found some related papers:
- https://robogen-ai.github.io/
- https://genesis-embodied-ai.github.io/, related HN thread: https://news.ycombinator.com/item?id=42457213
- https://paperswithcode.com/datasets?q=robot&v=th&o=match
These use other data sets, however.