[eru]

You are right about the technique not being very useful as of today. My fascination stems from my assumption that more resources poured into this approach would yield vastly better results.

Even just watching the video, I came up with several possible improvements to try out. Eg adversarial training, that would really hone in on the situations and aspects where the model is weak so far, like edge conditions; instead of just using normal gameplay as input.

[ehnto]

It's definitely an interesting area of research, but for that example you still had to make the whole game in the first place in order to have something to train the model on. Say you have a novel game idea, how could you use that approach to make it a reality? I'm not sure you could, but like you mention it's a really early example and who knows where it ends up.

The other part about that GAN Theft Auto example is that it doesn't actually know what's going on, like there's no game state. All it knows is that "When I have a frame that looks like this, and they press that button, I think the next frame would usually look like this". So it's got no internal game logic, it's just really good at painting what games look like.

[eru]

About the first one:

Even going about this very naively, you could at least use it to train a model against a supercomputer running the game, and then run the inference on much more modest end-user machines.

But you can be much more ambitious: have you seen eg style transfer? So you could probably do a bit of ML black magic to train your model on GTA, and then point it at the Google Earth data to get a GTA-like set in real-life London.

Or you could use something like style transfer to go for a cartoony look, or add ray-tracing like effects, even if you didn't have these effects in your original engine.

Or you can use a pre-trained model (eg on GTA), and then spend a relatively modest amount of extra training to get a different kind of game, eg one that has magic or so.

About the latter part: I do think their model is already running with some state. But even if it ain't, that's a relatively small thing to add with already known standard techniques (or you can come up with new techniques.)