Around 2015, a group of about a dozen data scientists at Amazon’s Edinburgh engineering hub began to notice something strange about the hiring tool they had spent a year developing. Men and women were not treated equally by the system, which was intended to scan resumes and rank applicants from one to five stars, just like Amazon customers rate products.
Without anyone telling it to, it had quietly taught itself that male candidates were just superior. Resumes with the word “women’s,” such as “women’s chess club captain,” were penalized. Graduates from all-women’s colleges were devalued. That was not programmed in. The algorithm faithfully absorbed and then amplified the data, which came from ten years of hiring decisions made by humans with their own patterns and preferences.
Key facts — AI recruiting tools & algorithmic bias
| Landmark case | Amazon AI recruiting tool (2014–2018) — penalized female applicants; scrapped by 2018 |
| Amazon tool bias mechanism | Downgraded resumes with “women’s” (e.g., women’s chess club); penalized graduates of all-women’s colleges |
| Companies using AI screening (2023) | 42% actively using; 40% considering adoption (IBM survey, 8,500+ IT professionals) |
| HR managers expecting AI adoption | 55% said AI would be regular part of their work within 5 years (CareerBuilder, 2017) |
| Notable individual case | Anthea Mairoudhiou (UK, 2020) — lost job after HireVue AI scored her body language poorly; HireVue removed facial analysis in 2021 |
| At-risk groups | Women, older applicants, minority groups, caste/religion (India context) |
| Major tools / companies involved | Amazon, HireVue, Goldman Sachs, LinkedIn, Hilton, Unilever |
| Core problem identified | Models trained on historically biased data reproduce and amplify existing discrimination patterns |
| Expert cited | Hilke Schellmann, NYU — author of The Algorithm: How AI Can Hijack Your Career |
| Reference / source | Reuters — Amazon scraps AI recruiting tool showing bias against women |
The project was terminated by Amazon. The company later stated that although recruiters consulted the tool’s recommendations, the tool was never used as the only decision-maker. The team had already attempted to modify the models to eliminate the gender bias by the time executives completely lost faith in the project. However, they discovered that correcting one issue did not ensure that the system would not discover new ways to sort candidates that might prove discriminatory. It’s worthwhile to consider the lesson hidden in that tale: you can correct a bias you can see. The ones that linger are the ones you can’t see.
2018 was that year. Since then, the technology has advanced significantly, and adoption has increased far more quickly than oversight. According to a 2023 IBM survey of over 8,500 IT professionals, 42% of businesses were already using AI screening tools in hiring, and another 40% were actively considering doing so.
Body language analysis, vocal evaluations, gamified cognitive tests, and CV scanners that analyze word choices and deduce fit from patterns the candidate was unaware were being measured are just a few of the techniques. The majority of these tools have an architecture based on historical data, which implies that they are, in a sense, using historical data to inform future decisions. Additionally, most industries were not particularly equitable in the past.
Once you hear about Anthea Mairoudhiou’s case, it stays with you. After being placed on furlough during the pandemic, the makeup artist from the UK was asked to reapply for her own job in 2020. An AI screening tool called HireVue, which examined candidates‘ body language during video interviews, was used as part of the procedure. On the skills evaluation, she received a good score. Her body language received a low rating from the AI. She was fired. The following year, HireVue quietly and without much public explanation discontinued its facial analysis feature. This was either a business decision or an admission that the feature had issues. It really is difficult to determine which.
The biggest risk AI poses to workers today is not that it will take their jobs, but that it will prevent them from getting one in the first place, according to Hilke Schellmann, an assistant professor at NYU who wrote a book analyzing these tools. This is something that more people in the industry should probably be saying aloud.
Observing businesses rush to automate the hiring process gives the impression that the promise of eliminating human bias, expanding the pool, and finding better candidates more quickly has somewhat outpaced the data. “We haven’t seen a whole lot of evidence that there’s no bias here,” Schellmann said, “or that the tool picks out the most qualified candidates.” That observation about the tools that currently stand in the way of millions of job seekers and the positions they are applying for is pretty damning.
Gender is not the only issue. According to research on AI adoption in India, systems developed without conscious consideration for representation run the risk of encoding biases pertaining to caste, religion, and socioeconomic status—replicating, through automated efficiency, the same exclusions that human recruiters have long been accused of. The claim that AI eliminates human bias begins to seem more like a rebranding than a solution in that situation. The prejudice persists. It is disguised as objectivity and laundered through mathematics.
The opacity involved makes all of this more difficult to handle. Theoretically, you can ask a human recruiter to explain why they rejected your application. An algorithm that evaluated your word choices based on ten years of past trends is unable to do so. It’s still unclear if any of the big players—Goldman, LinkedIn, HireVue, and the numerous smaller companies that supply screening software to mid-market HR departments—have created auditing procedures that are thorough enough to identify what they’re overlooking. Some businesses are making an effort. Given how widely these tools are now used, the question of whether trying is sufficient is quite different.
