This is kind of what tldraw are doing except they use 4V and instructions are png files. This could be a nice wysiwug for llamaindex or langchain agents in certain cases.
I really enjoy tinkering with LLM outputs that generate code that can be executed directly. Especially the faster models like GPT-3.5 Turbo are a joy to play with.