[RSZC]

There's a line between puff-up marketing and being actively misleading and OpenAI continually crosses it.

With only 25 heroes allowed (of > 100) they were unable to get >5k MMR...nowhere near pro level. What are they at with the full hero pool? 3k?

I'm not trying to cast aspersions on their AI work - it's obviously an incredibly hard problem. But instead of admitting that they failed to make a competitive team, they instead play a stripped down game that's easy for their bots' limited strategies, and then publish triumphant marketing releases claiming mission accomplished.

[Barrin92]

This seems to be a common occurence whenever new AI achievements pop up. A lot of "now that we've beaten xy" comments were also uttered around the starcraft match for example, but from the one game that was played without giving the AI a camera advantage it was clear that the play was still subpar.

The industry and researchers in particular should really be more careful with their claims, and honestly also try way harder to actually completely solve the challenges in question.

[anilshanbhag]

This is such a pessimistic view of things. As the hero pool expands, the training challenge increases at the rate of n^10. So going from 16 to 25 - we have ~100x blowup in hero combinations, likely requiring significantly more compute to train.

[RSZC]

  OpenAI Five is the first AI to beat the world champions in an esports game, having won two back-to-back games versus the world champion Dota 2 team, OG, at Finals this weekend.

That's their opening sentence, and it's actively misleading.

[cthor]

It's pretty sad people excuse this. Every single press release OpenAI have made in the history of this project has been actively misleading.

What they've achieved is actually impressive, too, so the deception is doubly annoying.

[usgroup]

... what do you suppose will happen as OpenAI transitions to being a more commercially oriented company with a fiduciary duty to its investors?

[wnevets]

A lot of pros don't have a hero pool that large. This sounds like the self driving car problem, AI has to be perfect despite the fact the humans are worse.

[Crespyl]

The bigger issue is the diversity of the pool, more so than the size.

True, any given team will tend to strongly favor a subset of heroes, but that subset by necessity must allow for a wide range of play styles and strategies.

The limited pool for OpenAI largely favors 5-man "deathball" tactics, which OpenAI plays fantastically well (an amazing achievement in its own right), but also excludes all the heroes best suited to countering that strategy. Sniper, for example, is an extremely strong hero in just the right circumstances, but is generally considered fairly weak and readily countered (in the current patch) by heroes such as Phantom Assassin or Spirit Breaker, neither of whom are present in the OpenAI pool.

Aside from anti-deathball specific counter picks, heroes that favor a wildly different approach to the overall strategy (split push/"rat" dota) such as Tinker are also excluded.

Most importantly, being able to work out what kind of strategy the enemy team is going for during the draft and adapting your own picks and bans accordingly (and luring your enemy into useless bans or counter-picks) is a huge part of the game. Once the draft is over, teams then have to be able to adapt their desired strategy to the reality of the team composition they landed on, and the best approach may change several times over the course of the game. I really want to see an AI that can realize that something isn't working in its current deathball strategy and that it needs to switch to playing both sides of the map at once by keeping the enemy occupied in once place while taking objectives in another (or vice versa).

To reiterate, I am incredibly impressed by what OpenAI has already shown, and I look forward to seeing where they go next, but I do think there are some very interesting problems in the full game that OpenAI has yet to demonstrate a solution for. I have similar slight disappointment with the recent AlphaStar StarCraft II project, which was a similarly amazing piece of work but also seemed to indicate that each agent (swapped out after each match with the human player) was only really able to execute one strategy and had no ability to recognize that something wasn't working and adapt (as seen in the final (and only) loss to the human player).

So, do I think that human pro players would defeat an OpenAI trained on the full unrestricted game in a best-of-three match? No, almost certainly not, but I do suspect that the humans would win in, say, a best-of-seven, if they had the full suite of tools available to allow them to discover and adapt to what I see as fundamental shortcomings of the AI.

[wnevets]

>So, do I think that human pro players would defeat an OpenAI trained on the full unrestricted game in a best-of-three match? No, almost certainly not, but I do suspect that the humans would win in, say, a best-of-seven, if they had the full suite of tools available to allow them to discover and adapt to what I see as fundamental shortcomings of the AI.

For now.

[Crespyl]

I'm optimistic that we'll figure out a solution in the long run, but I have yet to be convinced that any modern machine learning architecture will be able to get us there.

In essence, all of these game-playing models are playing by "instinct" (and playing very, very well), but still lack the ability to "jump out of the loop" to critique and adjust their own behavior on the fly. This is why the AlphaStar agent was vulnerable to being manipulated by the human players drop-harassment, and I believe OpenAI will demonstrate similar weaknesses once it opens to the public this week.

Machine learning has opened up all kinds of incredible advances that we can apply to the real world (see Boston Dynamics locomotion, or OpenAI applying the approach in their Dactyl robot hand project), but machine learning alone can't take us to general AI and I suspect it will always remain susceptible to this kind of deliberate manipulation.

[kevinwang]

They WERE at 5k. But this week they beat og.

[RSZC]

This is incorrect. From their post - they beat OG at 17 heroes in the pool. At 25 heroes in the pool they max'd at ~5k.