And here's[1] a project inspired by Gary's talk to design and implement a new escape protocol and a new virtual terminal to support these sorts of use-cases. (I'm not involved with the project.)
Isn't there a point of diminishing returns there? Also, isn't that a bit subjective? I'm glad VS Code exists, because it helps push other projects to be better.