I first tried out something like this in the early 1990s at VPL Research - one of the originators of the VR industry. They had a visual programming language called Body Electric for controlling how things behaved in VR. Some details here: https://news.ycombinator.com/item?id=22788773