The Biased Algorithm
2 min
AI Resume Screener: Analyze candidate
In 2018, a major tech company discovered that the AI hiring tool it had been developing for four years was systematically penalizing resumes that contained the word "women's", as in "women's chess club" or "women's college." The system had learned from a decade of historical hiring data, which reflected the industry's existing gender imbalance. Resumes from male candidates had led to more hires, so the AI concluded that male candidates were simply better. Nobody caught it for years because the system appeared to be working. It was efficient, consistent, and confident in its recommendations. But efficiency without fairness is not progress. This raises a critical question: if an AI system produces biased outcomes but nobody checks, does the bias exist? And whose responsibility is it to check?
An AI hiring tool that discriminated against women, and nobody noticed.