[crispyambulance]

One thing I've always wondered about these automated game-playing applications is what they use as input.

Are they just getting images (frame by frame) as input, perceiving them in some way, and producing an output stream of controller button-pushes?

Or does it only detect "death" and start over, systematically changing it's robotically timed output stream in response with the goal of getting further along in the game?

I know the retro games were very deterministic. Pacman had "patterns" you could use. Was Mario like this? No random adversaries at all?

[cfiggers]

I have seen systems that use frames/screen captures as input (look up sentdex' GTA series on YouTube for an example) but that isn't what's happening here. In this case, it looks like the `dist` function is directly accessing key game values by reading bytes out of the emulator's memory. So the AI is taking multiple signals from the game's state as input, not just "alive/dead."

The blog post explains that part a bit under the "Distance" section.

[matheusmoreira]

They're reading the game's state from memory. This is also commonly done in tool assisted speedruns.

> Was Mario like this? No random adversaries at all?

Yeah. Read all about it here:

https://tasvideos.org/GameResources/NES/SuperMarioBros

https://tasvideos.org/GameResources/CommonTricks#LuckManipul...

https://tasvideos.org/GameResources/CommonTricks#ExamineGame...

https://tasvideos.org/LuckManipulation