← Insights
Recruitment & bias

The first recruiter isn't human

Automated CV screening is where artificial intelligence enters HR processes first, and most often. What it really does, why the cost falls on the employer and not on the vendor, and what EU law already requires. Today, not in 2027.

Take a CV. Leave everything identical (experience, skills, qualifications) and change only the name at the top of the page. From Mark to Fatima. From Julia to John. In many automated screening systems the score changes. And whoever published the vacancy never notices.

This is not a hypothesis. From 2014 until early 2017 Amazon worked on a tool that gave candidates a score from one to five stars, much like the products on its marketplace. Trained on ten years of CVs it had received, mostly from men because the tech industry is mostly male, the system learned that men were preferable: it penalised CVs containing the word "women's" (as in women's chess club) and downgraded graduates of two all-women's colleges. Amazon tried to fix it, could not make it reliably neutral, and abandoned it. The point is not that the tool was defective. The point is that it did exactly what a system of this kind does by default: learn from the past and repeat it.

That past is now almost everywhere. Automated screening is the first, and often the only, filter a CV meets before a human reads it. In the United States, nearly all Fortune 500 companies use an applicant tracking system: 97.4% in 2026, according to Jobscan's analysis of their career sites. What matters is another feature: this first reader is invisible, fast and not human, and in many cases it ranks or rejects an application before anyone looks at it.

Same CV, different name

The question is not whether these systems make mistakes, but how measurable the mistakes are. One of the largest recent tests comes from a University of Washington study presented in October 2024 at the AAAI/ACM Conference on AI, Ethics, and Society. The researchers took more than 550 real CVs, applied to them 120 first names associated with white and Black men and women, and asked three language embedding models (the same kind of technology that, in screening software, measures how closely a CV "resembles" a job advert in order to rank candidates) to rank those CVs against more than 500 real job listings. Over three million comparisons.

The result, measured across 27 tests: the models favoured names associated with white candidates in about 85% of tests, and in no test did they prefer a name associated with Black men over one associated with white men, for the same CV. One detail makes the problem more uncomfortable for anyone who believes they have solved it: removing the name is not enough. Indirect signals (word choice, the name of a school, a student society) still point to identity, and the authors note that models are likely to pick them up too.

None of this started with AI. As early as 2004, a well-known field experiment by Bertrand and Mullainathan sent fictitious CVs to real job advertisements, randomly assigning white- or African-American-sounding names: those with "white-sounding" names received about 50% more callbacks. Selection bias is an old human problem. AI did not invent it: it inherited it from historical data and applied it on an industrial scale, at almost zero marginal cost and without leaving a visible trace.

It isn't the software being "bad"

Precision matters here, because this is what separates a serious analysis from technophobic alarm. These systems have no discriminatory intent. They produce distortions in two distinct ways, and both lead back to human choices.

The first is the training data. A model that learns from unbalanced past hiring learns the imbalance: that is exactly what happened at Amazon. The second, less discussed but more widespread, is configuration. The study Hidden Workers: Untapped Talent by Harvard Business School and Accenture (2021), based on a survey of more than 2,250 executives in the United States, the United Kingdom and Germany, found that 88% of employers agree that qualified high-skills candidates are vetted out because they do not match the exact criteria in the job description, a figure that rises to 94% for middle-skills candidates. The same study estimates that there are about 27 million "hidden workers" in the United States alone: people who want to work, or to work more hours, but are held back by the way companies hire. According to the study's authors, for example, about half of US companies filter out applicants with a gap in their work history of more than six months.

In other words: the problem is often not "the bad algorithm", but a selection rule poorly defined by a human and then applied rigidly, across thousands of applications, by a machine that does not argue. The filter does what it was set up to do. It is invisible rigidity, not malice, that causes the harm.

Why the cost falls on the employer, not on the vendor

For a company the cost of all this is twofold, and the vendor pays neither part.

The first cost is a business cost. In a labour market where almost everyone complains of skills shortages, a filter calibrated on exact requirements narrows the pool until it excludes people who would do the job very well. Companies end up searching for a long time for candidates who are "perfect on paper", while capable candidates are rejected at the second decimal place of a score. It is a talent shortage that is partly self-inflicted.

The second cost is legal, and this is where responsibility shifts clearly to the company that uses the tool, not to whoever sold it. Relying on a model supplied by a third party offers no protection against the consequences of a discriminatory outcome: the decision remains the employer's.

What the law already says, and it isn't the AI Act

There is a widespread misconception: that the issue is governed by the AI Act, and that there is therefore time. The opposite is true. Under the AI Act (Regulation (EU) 2024/1689), AI systems intended to be used for recruitment or selection, "in particular to place targeted job advertisements, to analyse and filter job applications, and to evaluate candidates", are classified as high-risk (Annex III, point 4(a)). The obligations for these systems have been postponed to 2 December 2027 by Regulation (EU) 2026/1744, the so-called Digital Omnibus on AI, published in the Official Journal of the European Union on 24 July 2026 and in force since 27 July 2026. From that date, among other things, employers deploying such a system will have to inform workers' representatives and the affected workers before using it (Article 26(7)). For recruitment systems, then, the specific AI Act obligations are the next layer of regulation. They are not what binds employers today, apart from general duties already in force, such as AI literacy (Article 4).

What binds employers today is already fully in force, at several levels.

EU equality law. Three directives prohibit discrimination in "conditions for access to employment", expressly "including selection criteria and recruitment conditions": Directive 2000/43/EC on racial or ethnic origin, Directive 2000/78/EC on religion or belief, disability, age and sexual orientation, and Directive 2006/54/EC on sex. All three define indirect discrimination: an "apparently neutral provision, criterion or practice" that puts a protected group "at a particular disadvantage", unless it is objectively justified by a legitimate aim and the means of achieving it are appropriate and necessary. An automated filter that produces a disparate impact does not become lawful because a machine applies it. And all three shift the burden of proof: once a candidate establishes, before a court or other competent authority, facts from which discrimination may be presumed, it is for the employer to prove that there has been no breach of the principle of equal treatment.

The GDPR, Article 22. Under Article 22(1), a person has the right "not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her". Where such a decision is allowed because it is necessary for entering into or performing a contract, or on the person's explicit consent, the controller must provide safeguards, "at least the right to obtain human intervention on the part of the controller, to express his or her point of view and to contest the decision" (Article 22(3)). In the SCHUFA case (C-634/21, 7 December 2023), which concerned credit scoring, the Court of Justice of the EU clarified a decisive point for anyone who thinks a "human in the loop" is enough: an automatically generated score is itself an automated individual decision where the party receiving it "draws strongly" on it to establish, implement or terminate a contractual relationship. Applied to recruitment (the extension is by analogy, but the principle is the same): if the rejection of a candidate is in practice determined by the system's score, and human review is only a formality, Article 22 applies.

That brings duties of transparency. The person concerned is entitled to "meaningful information about the logic involved" (Articles 13(2)(f) and 15(1)(h)), and in Dun & Bradstreet Austria (C-203/22, 27 February 2025) the Court held, on the right of access (Article 15(1)(h)), that this means explaining "the procedure and principles actually applied" to that person's data, in a concise, transparent, intelligible and easily accessible form. Handing over the algorithm is not enough. Where the processing is likely to result in a high risk to people's rights and freedoms, a data protection impact assessment is also required before it starts (Article 35). And Article 88 allows Member States to adopt more specific rules for the employment context, expressly including recruitment.

Pay transparency. Directive (EU) 2023/970, which Member States had to transpose by 7 June 2026, adds two rules that apply to recruitment even when no algorithm is involved, but which often live inside recruitment software. Applicants have the right to receive information about the initial pay or its range, for example in the published vacancy notice (Article 5(1)). And: "An employer shall not ask applicants about their pay history during their current or previous employment relationships" (Article 5(2)). Many application forms still ask for current salary, sometimes as a mandatory field. In practice the obligation applies through each country's implementing law, which is worth checking wherever you hire. It is also directly relevant to the subject of this article: a candidate's pay history is one of the data points through which a past pay gap can find its way back into today's selection.

National rules may go further. Italy is an example. Article 1-bis of Legislative Decree 152/1997, introduced in 2022 and limited in 2023 to fully automated systems, requires employers to inform workers and trade union representatives about the use of fully automated decision-making or monitoring systems, including those used for hiring, and to explain their logic and functioning, the categories of data and main parameters used to programme or train them, and the potentially discriminatory impacts of their metrics (subject to an exemption for systems covered by trade secrets). At EU level, Directive (EU) 2024/2831 introduces similar transparency duties on algorithmic management, but only for digital labour platforms, with a transposition deadline of 2 December 2026.

The risk is not theoretical. In November 2024 the Italian data protection authority fined Foodinho, part of the Glovo group, five million euros for unlawfully processing the personal data of more than 35,000 riders. Among other infringements, it found that the company used automated systems without the safeguards required by the GDPR: the right to obtain human intervention, to express one's point of view and to contest the decision. The case concerns work managed through a digital platform, not CV screening. But the principle is the same and transferable: an automated decision about people, in the context of work, requires safeguards, genuine human oversight and the absence of discrimination. And regulators are already enforcing it.

How to look inside your own black box

The answer is not to give up the tool. It is to stop treating its output as objective, and to check it. Five practical checks, which do not require a data science team.

  1. The name test. Take 15 to 20 real CVs, duplicate them changing only the name (different gender and background), run them through the system again and compare the rankings. If the order changes for the same content, the problem is in-house.
  2. Five questions for the vendor, in writing. Which variables does the model use to score? Has it been audited for bias, and with what results? Can it explain why a candidate was rejected? What data was it trained on? Can an operator override its outcome, and how?
  3. A map of exclusion rules. Which filters are hard constraints (exact job title, exact years of experience, a specific degree, career gaps) and which are mere preferences? Every hard constraint is a point where suitable candidates are being rejected without anyone knowing.
  4. A sample of rejections. Pull 50 CVs rejected automatically and have a person review them. How many were genuinely unqualified? The difference is the hidden cost of the filter.
  5. Human oversight, documented. Human intervention must be real, not a rubber stamp, and it must be recorded. It is both the best defence against a discriminatory outcome and the evidence that the risk has been managed.

Automated screening was born with a promise: to remove human prejudice from selection. In practice, too often, it has done the opposite: it has made it faster, broader and harder to see. The remedy is not to go back to reading thousands of CVs by hand. It is to recognise that a score produced by a machine is not a neutral fact: it is a decision. And decisions about people, especially when they are made by a system nobody has looked inside, need someone who is accountable for them. Today, not in 2027.

Probum AI

These checks, applied to your company and put in writing.

Probum AI, based in Italy, documents the logic, safeguards and human oversight of the automated systems a company uses on its people, in a signed compliance dossier.

Write to us

Sources

  1. University of Washington: K. Wilson, A. Caliskan, Gender, Race, and Intersectional Bias in Resume Screening via Language Model Retrieval, AAAI/ACM Conference on AI, Ethics, and Society, October 2024 (arXiv v3, 30 August 2026, with erratum on the gender-only results).
  2. Jobscan, 2026 Applicant Tracking System (ATS) Usage Report, 20 August 2026.
  3. M. Bertrand, S. Mullainathan, Are Emily and Greg More Employable Than Lakisha and Jamal?, American Economic Review, 2004.
  4. Reuters (J. Dastin), Amazon scraps secret AI recruiting tool that showed bias against women, October 2018.
  5. Harvard Business School & Accenture, Hidden Workers: Untapped Talent, 2021.
  6. Regulation (EU) 2024/1689 (AI Act), Annex III, point 4(a), and Article 26(7); Regulation (EU) 2026/1744 (Digital Omnibus on AI), OJ L of 24 July 2026, in force since 27 July 2026 (high-risk obligations for Annex III systems applicable from 2 December 2027).
  7. Council Directive 2000/43/EC, Articles 2(2)(b), 3(1)(a) and 8(1); Council Directive 2000/78/EC, Articles 2(2)(b), 3(1)(a) and 10(1); Directive 2006/54/EC, Articles 2(1)(b), 14(1)(a) and 19(1).
  8. Regulation (EU) 2016/679 (GDPR), Articles 13(2)(f), 15(1)(h), 22, 35 and 88.
  9. Court of Justice of the EU, Case C-634/21, SCHUFA Holding (Scoring), judgment of 7 December 2023.
  10. Court of Justice of the EU, Case C-203/22, Dun & Bradstreet Austria, judgment of 27 February 2025.
  11. Directive (EU) 2023/970 (pay transparency), Articles 5 and 34.
  12. Italy, Legislative Decree 152/1997, Article 1-bis (introduced by Legislative Decree 104/2022 and amended by Decree-Law 48/2023).
  13. Directive (EU) 2024/2831 (platform work), Articles 1, 9 and 29.
  14. Garante per la protezione dei dati personali (Italian data protection authority), decision on Foodinho S.r.l. of 13 November 2024 (fine of EUR 5,000,000).