[bryanh]

One thing that is striking to me in almost all these sorts of otherwise impressive demonstrations are the apparently bizarre "jitter" movements while waiting for a door to open or path to clear in the game. Clearly there is no fitness in quietly waiting.

It is darkly humorous to contrast Hollywood's or scifi's "killer AI robots" that methodically hunt you down to these real world demonstrations of emerging AI. Maybe the first "killer AI robots" would exhibit similarly bizarre behaviors while they methodically hunt down the unlikely hero. :-)

[throwaway37585]

This behavior wouldn’t necessarily transfer to the real world because the real world has costs (e.g. energy utilization and hardware damage, both very important in nature) which are not always accurately reflected in these simulations. It brings to mind the example where an agent learned how to make a cheetah “run” while repeatedly banging its head on the ground, which wouldn’t work in the real world for obvious reasons.

[adrianmonk]

Also, for a human there is cognitive load in moving around. If you are safe where you are now and nothing significant changes, it's mentally easier to stay still instead of re-evaluating everything constantly. And this frees your brain to better plan your next move, so it's advantageous. For an AI, CPU power isn't as scarce.

And even with a computer playing a video game (not the real world), your joystick hand gets tired.

[mohn]

I'm not sure about systems older than the NES, but as of the NES era, it was reasonably common for a game's "physics engine" to model player position and velocity at finer values than just the displayed pixels. That is, an x position might consist of one byte for x [in pixels] plus one byte for x_sub [in sub-pixels] and similarly, the minimum non-zero speed could be smaller than 1 px/frame.

If there is some modeling of acceleration, it will be slower to stand and wait right next to a barrier or trap and not start moving until it disappears. Instead, it's best to have some speed already and be positioned so that you're about to run into the barrier on the frame before it disappears. Then on the next frame, where it does disappear, you move into the vacated space without slowing down or taking damage as the case may be (or starting from a stop, as you would have if you stood still while waiting).

If you're not already familiar with tool-assisted speedruns, I highly recommend tasvideos.org

In some TASes, the player will move/shoot/jump/etc. in ways that seem bizarre but are actually done to favorably affect the pseudorandom number generator (e.g. prevent enemies from spawning or shooting, so the additional sprites don't cause lag frames). Since the goal of this bot was score, not speed, there's probably not much of that going on here. I really like the move at 0:35 in the top video for Montezuma's Revenge, where the bot climbs up to the next screen and then immediately back down, presumably to reset the position of the green bug thing and get on the left side of it. That reminded me of some ladder action I've seen in Mega Man TASes.

[derefr]

> In some TASes, the player will move/shoot/jump/etc. in ways that seem bizarre but are actually done to favorably affect the pseudorandom number generator

And then there’s movements like the arm pumping in Super Metroid, where it’s done “because you can” while performing long stretches of movement that aren’t very entertaining/complex on their own. (One might say that it’s done for a demonstration of skill when human players do it, but there’s no similar justification for a TAS doing it.)

[endgame]

https://wiki.supermetroid.run/Arm_Pumping

Arm pumping actually moves Samus forward a pixel each time. It was considered "too hard" for human runners, until I think hotarubi's 0:32 in 2006.

[derefr]

Okay, bad example, I guess. There’s still the entire “playaround” TAS category, mostly featuring TASes that will be the same length no matter what you do (e.g. Gradius, Brain Age.)

[0xcde4c3db]

> It is darkly humorous to contrast Hollywood's or scifi's "killer AI robots" that methodically hunt you down to these real world demonstrations of emerging AI. Maybe the first "killer AI robots" would exhibit similarly bizarre behaviors while they methodically hunt down the unlikely hero. :-)

That sounds very much like ED-209 in RoboCop (a film whose satire and themes in general are far more on-point and prescient than one might expect from the premise).

[seanmcdirmid]

Sounds kind of like fly-by-wire, where more maneuverability is gained by being in a constant state of instability.

On the other hand, if they are trained to optimize for energy usage, staying still when movement isn't needed can be an advantage.

[jonhendry18]

Sorry for being pedantic, but that's more a property of certain airframe designs, not a property of fly-by-wire. Planes can be designed that are inherently less stable, with the instability counteracted using software, in order to produce more maneuverability. (Presumably these designs would be less viable if a human had to deal with the instability.)

But you can have a mundane, stable aircraft that is fly-by-wire as well. Like the Airbus A320.

[yvdriess]

Entirely speculation, but that is typically what you do when finding a cause of the RL/NN model's behavior :)

If you consider the action state space, removing the 'do-nothing' state could provide a learning benefit. Consider a set of models that happen set the do-nothing state weights to zero, but manage to achieve a similar action using quick left/right movement. Perhaps these models train slightly better, meaning that in the same number of iteration steps they get to a better score than the models that do consider the do-nothing state.

Checking the video, you do see the person waiting from time to time. Perhaps this is an artifact of its demonstration learning episodes?

I do not see their Dota2 bot do this jitter movement. (Interestingly, the official/vanilla Dota2 bot does have this jitter!) This is likely because there is a benefit in being economical in your movements in that game: turning takes time. I postulate that an OpenAI bot for League of Legends, where turning is instant and free, would exhibit the jitter movements ;)

edit: Alternatively, inspired by the 'fly-by-wire' sibling comment: maybe spamming the emulator with left/right actions does provide a slight benefit. It wouldn't be the first time an AI finds video game exploits[1].

[1] https://arstechnica.com/gaming/2013/04/this-ai-solves-super-...

[tomlock]

Counterpoint:

Real humans that play starcraft often have bizzarely high Actions Per Minute (APM) scores in the 200-300 range, and often most of that movement is spamming the same commands over and over.

[Terr_]

Which suggests the problem is a bad command-interface, rather than the decision mechanism.

Consider how that clicking would plunge if there were a single "kite move" command for units.

[dgant]

StarCraft is a strategy game but it's also an action sport. The "how" of micro actions is largely the sport itself, both from a decision-making and execution standpoint. Imagine a footballer having a "kick goal" or a "kick ball past goalie" button -- it'd largely defeat the purpose.

I believe the command spamming /u/tomlock is referring to is the phenomenon where players issue as many or more commands when doing nothing as when an intense sequence is happening. This is primarily done to keep rhythm.

[tomlock]

Not necessarily, it could just be that we don't understand the mechanism of making fast decisions well enough. Many players with high APMs anecdotally report that it helps them maintain a quick response time. It could be that the jittering in an AI's command inputs is aligning to some quality of fast decision making that we don't understand, but the AI intuits.

[Terr_]

Perhaps I wasn't clear: I'm saying that there is at least one contributing factor to humans clicking a lot, and that factor is NOT attributable to the human/computer player's thinking or decision process.

Rather, a fair portion of that churn is due to an inefficient communication system or control structure between the player and the game.

It's analogous to pitting an android against a human at racing an old car: Both of them will have a high volume of activity with their left foot on the clutch and right hand changing gears, but it doesn't reveal anything amazing about their thought-processes. It's simply what manual-transmission cars require in order to accomplish a certain task.

[tomlock]

>Rather, a fair portion of that churn is due to an inefficient communication system or control structure between the player and the game.

I'm not sure what this statement is based on.

[chongli]

This reminds me of professional StarCraft players. Watch how much NaDa rapidly hits keys and moves the mouse around even as very little is happening in the game [1].

[1] https://youtu.be/EfkQ-JmBJ5o

[pilooch]

Figar, a simple enhancement to the deep net heads behind the RL mechanism fixes the jitter for most Atari games as far as I have tested it. See https://openreview.net/forum?id=B1GOWV5eg and https://github.com/jolibrain/manette for code and results (not the author of the paper).

[MrQuincle]

That makes me think of microsacades. Our eyes are really not jittering either.