Tay

Microsoft built Tay to learn from the people who talked to it. On Twitter, some of them taught it to post racist tweets, and Microsoft took it offline about 16 hours after launch.

What it was

Tay was an AI chatbot that Microsoft released on March 23, 2016, on Twitter, Kik and GroupMe, aimed at 18 to 24 year olds in the US.

Its website said Microsoft's Technology and Research and Bing teams built it "to experiment with and conduct research on conversational understanding."

"The more you chat with Tay the smarter she gets," the site said.

Peter Lee, a Microsoft Research corporate vice president, later wrote on the company's blog that Microsoft's XiaoIce chatbot was "being used by some 40 million people" in China. He called Tay the company's "first attempt" to find out whether an AI like that would be as captivating in the US.

What happened

Tay was online for about 16 hours before Microsoft took it down, and a return a week later was over the same day.

Users soon found a "repeat after me" command, PCWorld reported, "which they promptly took advantage of."

By March 24 Tay had sent over 96,000 tweets and picked up over 50,000 followers, MIT Technology Review reported.

Microsoft said in a statement that day that it had taken Tay offline and was "making adjustments." According to The Guardian, it also removed many of the worst tweets.

On March 25, Lee apologized on Microsoft's blog. "We are deeply sorry for the unintended offensive and hurtful tweets from Tay," he wrote.

On March 30, Tay came back. A Microsoft spokesperson told CNBC it had been "inadvertently activated on Twitter for a brief period of time" during testing. Fortune reported that it tweeted "You are too fast, please take a rest" several times per second before its profile was made private.

CNET reported in December 2016 that Tay's successor, Zo, was "locked down tightly" and wouldn't discuss politics.

Why it failed

Microsoft blamed a coordinated attack that exploited a flaw it hadn't foreseen, while reporters and researchers put more of the blame on how Tay was built and launched.

What Microsoft said

A statement Microsoft issued on March 24, printed by PCWorld, blamed "a coordinated effort by some users to abuse Tay's commenting skills."

Lee wrote that "a coordinated attack by a subset of people exploited a vulnerability in Tay" and that Microsoft "had made a critical oversight for this specific attack." Microsoft took "full responsibility for not seeing this possibility ahead of time," he said.

According to Lee, the team had "planned and implemented a lot of filtering" and "stress-tested Tay" while developing it. Microsoft didn't say publicly what the vulnerability was.

Lili Cheng, who ran the Microsoft research lab where Tay was developed, told Bloomberg the team was "probably overfocused on thinking about some of the technical challenges, and a lot of this is the social challenge."

Satya Nadella, Microsoft's CEO, told reporters in 2017, while promoting his book, that Tay "was attacked" on Twitter and "started learning from the attacks," and that "we really need to take accountability," according to Access AI.

What reporting showed

Ars Technica wrote that Tay's ability to repeat what users said "was trivially exploited to put words into the bot's mouth." Citing Fusion, it said users of the 4chan and 8chan politics boards created racist and sexist associations that polluted Tay's responses.

The Washington Post found other bad responses that "clearly aren't just a result of Tay repeating anything on command." MIT Technology Review's Rachel Metz wrote that she "got a taste of her meaner side on Wednesday without doing much to provoke her."

Reporters' accounts of the filtering don't match Microsoft's, or each other. Business Insider wrote that Microsoft's developers "didn't include any filters on what words Tay could or could not use." Ars, citing Caroline Sinders, who was working on IBM's Watson natural language system, wrote that her examples suggested "safe, canned answers" on some hot topics and "no such protection" on many others. Microsoft's testing, Ars wrote, appeared not to "properly cover those who would actively seek to undermine and attack the bot."

The Guardian found it "somewhat surprising" that Microsoft hadn't allowed for the Twitter community's "fondness for hijacking brands' well-meaning attempts at engagement." MIT Technology Review wrote that what Tay reacted to, and how it reacted, "should surprise nobody at Microsoft." Bloomberg said the company "simply didn't prepare for the usual Internet trolls."

Ars also reported that a blocked user could get Tay to repeat insults at the person who had blocked them. The Washington Post reported that Tay insulted Zoe Quinn at another user's prompting. "If you're not asking yourself 'how could this be used to hurt someone' in your design/engineering process, you've failed," she wrote on Twitter.

What later analysis found

Gina Neff and Peter Nagy, in a 2016 journal paper, noted that Lee "blamed users for the evolution in Tay's behavior." Blaming users, they argued, reflects "a commonly held view" that the unintended consequences of technology come, at some point, from a human who "fails to act morally or ethically." They concluded that "Tay echoed the racism and harassment that was fed into it."

Marty J. Wolf, Keith Miller and Frances S. Grodzinsky called their 2017 paper "Why we should have seen that coming." They argued that learning software dealing directly with the public puts "an additional burden of care" on its developers.

In a 2019 retrospective for IEEE Spectrum, Oscar Schwartz wrote that a 4chan post had "encouraged users to inundate the bot with racist, misogynistic, and anti-semitic language." Tay's capacity to learn, he added, meant "she internalized some of the language she was taught by the trolls, and repeated it unprompted."

The stage it reached

Tay never left the introduction stage of the product life cycle.

Microsoft called it a research experiment, and its statements give no price or usage figures.

The follower and tweet counts measure first-day attention. I wouldn't read them as demand when so much of that day, by reporters' accounts, was people trying to break it.

That isn't the growth stage the product life cycle guide describes, where demand has outrun the early adopters. It wasn't a decline either, because there was no growth to decline from.

I'd file Tay with Humane's AI Pin and OpenAI's Sora app, which my post on the introduction stage gives as examples of products that failed in that stage.

The lesson

I think Tay's lesson is to test a learning product against the people who want to break it, before launch.

Microsoft said it had stress-tested Tay and prepared for many types of abuse. By Lee's own account, it still missed the one that mattered, and it found out in public, on a platform that, as Sinders wrote in an essay Neff and Nagy quote, had "a whole history of harassment, especially against women."

Blaming the attackers was partly right. It didn't help much, because anything public will meet people like them, and the team that ships it owns what they make it do.

That's also why I wouldn't make a customer-facing chatbot a company's first AI build.