[resu_nimda]

Unfortunately, in the first 24 hours of coming online, a coordinated attack by a subset of people exploited a vulnerability in Tay. Although we had prepared for many types of abuses of the system, we had made a critical oversight for this specific attack.

Well that's total BS. Releasing a thing like this on the open internet without a simple "don't say Hitler" rule? It had a feature where it would repeat anything you say. Abusing that doesn't require a sophisticated coordinated attack, as they imply. What kinds of abuse did they prepare for, then?

This is a colossal failure to demonstrate a basic understanding of how (some) people act on the internet. I just don't know how they expected anything other than this exact outcome.

[taf2]

I'm by no means someone who would normally defend Microsoft but for real we are all learning. Failure is a successful outcome of research. Discovering vulnerablities is a valuable outcome.

[resu_nimda]

It's funny because I'm in the camp that has been impressed with Microsoft lately. And of course it's ok to make mistakes, even really big ones.

But I would not say that this failure was a successful outcome, at least not nearly as successful as it could have been. They had to shut it down within hours and all we really learned is that people on the Internet like to troll with incredibly offensive stuff. Most of us knew that, I'm pretty sure. If they had actually prepared for abuse we might have learned more interesting things.

What really gets me though, is just the obnoxious spin on the press release implying that they had prepared so well for abuse but a sophisticated coordinated super-hacker attack found the tiny vulnerability.

[oneeyedpigeon]

I, too, want more details. What evidence does Microsoft have that the 'attack' was coordinated? What evidence do they have that it was an 'attack' at all? Is the 'vulnerability' they refer to merely the part of the algorithm that parrots input? That's not a vulnerability, that's a core function of the software.

More interestingly, at what point does repeating my opinion (however heartfelt, misguided, or unpopular) become a 'coordinated attack'?

[tacos]

Chatbots have been around for fifty years. Getting them to say dirty things has been a sport since 1964.

https://en.wikipedia.org/wiki/ELIZA

Launching this on the open internet in this state was pure stupidity. Doing it on Twitter during an election cycle borders on "career-ending."

[joe_the_user]

Well,

Claims that MS did due diligence aside, Tay having a vulnerability that some people exploited is still better explanation than "it was a real AI that learned poor morals from the Internet".

Also, the claim was that there were small groups that input a lot of data to the bot quickly - the script might have been "don't say X unless everyone is saying X already", which might have worked in small tests but clearly could be exploited.

I suspect most talking as if "she" was like a teen soaking up random bad ideas are falling for the Eliza Effect.

https://en.wikipedia.org/wiki/ELIZA_effect

[mizzao]

When your AI system is data-driven, you can't simply filter out words and expect not to have oversights. Something will always be overlooked, and someone on the Internet will inevitably say "that was obvious".

For example, consider the person who sent Tay an image of Hitler and it sent back a circle around his face, labeled "So swag". Is Tay supposed to know what Hitler looks like too? Is it supposed to be able to recognize other, much more subtle images?

In all these cases the bot learns from data, not rules, and the social problem is that we can't label data as moral or immoral, right or wrong.

[resu_nimda]

Well this gets into the much deeper issue that it's not actually an AI, it's not actually "learning." It's taking things people say and mixing them up and repeating them (someone's going to claim that that's exactly what people do, but I think it's clear that this bot would never be able to gain "true understanding" of words and meaning).

In general, I think the recent wave of interest in chatbots/personal assistants is premature, these ones are really no smarter than SmarterChild, because we haven't gotten any closer to building a system that can actually assign true "meaning" to arbitrary language.

The data has been modeled, cleaned and filtered by the team who built Tay, the bot’s website states.[1]

1. They did a lot of manual massaging of the data anyway.

In addition, if a user chooses to share with Tay, it will track their nickname, gender, favorite food, zip code and relationship status as part of its personalization.[1]

2. These features seem hardcoded.

Meanwhile, its responses – which are meant to be funny, not serious – were developed by a staff which included improvisational comedians, says Microsoft.[1]

3. It also uses scripted content. They imbued it with a totally fake personality. They made it mimic a human, it uses slang that was programmed into it.

It's a hodgepodge of techniques that to me are largely "faking it." So I don't buy the idea that they just sort of turned on this learning engine and had no control over its "morality."

Microsoft knew all of this, they knew it wasn't actually learning language. So what is the purpose of releasing it to talk to people on Twitter? Basically a PR stunt, to get people to have fun with it. From that perspective, a really simple blacklist of words would have gone a long way, and not compromised the integrity of its "learning" (because it was already entirely compromised). And yeah, I mean, a bit of image recognition doesn't seem out of the question either (not saying it would have prevented the bot from repeating/reposting "bad" things, but it maybe would have had some chance to develop a narrative other than "so racist and offensive it was immediately shut down").

Yeah I still do think it is obvious that you shouldn't release a fake AI bot that will blindly repost images that are sent to it by Twitter users, and that there is nothing really to be gained by doing so.

[1] http://techcrunch.com/2016/03/23/microsofts-new-ai-powered-b...