[chongli]

I think it would just take a ton of work. If you’re not willing to make the leap to AI (and non-determinism) then conventional technology doesn’t have much to offer for NLP (which is what the task ultimately boils down to).

The trouble with trying to build a deterministic, logic-based NLP for interacting in a game world is essentially a special case of the frame problem [1] in AI.

[1] https://en.wikipedia.org/wiki/Frame_problem

[fl7305]

I'm guessing it would be like a fever dream to let an LLM be the dungeon master in isolation. Things would change after the fact in weird ways, especially when the context grows too large.

But what about coupling an LLM with a physics simulator and a 3D world model?

You still interact with the LLM in a text interface, but hidden conversations take place with the simulator where the LLM can interrogate the current state of the 3D world simulator to describe it to the player. You could even do this using GPT4-Vision to interpret rendered images. When the player performs an action, it is translated into "physical" actions into the 3D world simulator which updates its state.

It feels like someone should have done this already?