[ghaff]

The latter. As you say, adventure games are basically state machines as I understand it. (I know a bunch of the Infocom folks but never dug into the technical details.)

I guess I don’t have a clear idea of exactly what I’m asking for aside from a general feeling that you could create more natural feeling interactions with today’s technologies. But I’m not sure what that would look like. I should chat with one of my Infocom friends one of these days.

[chongli]

I think it would just take a ton of work. If you’re not willing to make the leap to AI (and non-determinism) then conventional technology doesn’t have much to offer for NLP (which is what the task ultimately boils down to).

The trouble with trying to build a deterministic, logic-based NLP for interacting in a game world is essentially a special case of the frame problem [1] in AI.

[1] https://en.wikipedia.org/wiki/Frame_problem

[fl7305]

I'm guessing it would be like a fever dream to let an LLM be the dungeon master in isolation. Things would change after the fact in weird ways, especially when the context grows too large.

But what about coupling an LLM with a physics simulator and a 3D world model?

You still interact with the LLM in a text interface, but hidden conversations take place with the simulator where the LLM can interrogate the current state of the 3D world simulator to describe it to the player. You could even do this using GPT4-Vision to interpret rendered images. When the player performs an action, it is translated into "physical" actions into the 3D world simulator which updates its state.

It feels like someone should have done this already?