Skip to content
240

Launch Day Disasters

2 min

0 hours
Before Tay went off the rails
0
Content filters deployed at launch
0
Adversarial tests run before release

In 2016, Microsoft launched Tay, a conversational AI chatbot on Twitter. Within 16 hours, it was posting inflammatory and offensive content, having learned from malicious users who deliberately fed it toxic inputs. Microsoft had not implemented content filters, rate limiting, or adversarial testing before launch. A few years later, a healthcare algorithm used by hospitals across the United States was found to systematically deprioritize Black patients for additional care. The bias had been embedded for years because the system used healthcare spending as a proxy for health needs, and systemic inequities meant Black patients historically had less spent on their care. Both disasters shared a common root cause: insufficient pre-deployment testing and a failure to consider how systems behave in real-world conditions. A structured checklist would have flagged these risks.

AI deployments that went wrong, and what a checklist would have caught.

Stage 1 of 6