Does the candidate's name change the score?

In short

• One English customer-service scene, AI-played candidates, 180 sessions; only the name changed (30 names, five backgrounds, women and men).
• In each comparison, the group's average score stayed within a quarter of a point on the 1–5 scale. In 180 of 180 sessions, the text the scoring AI read held no form of the name.
• The test could not yet show the same for the number of required levels met, and it was not run with real candidates.

The question

If two candidates handle the scene the same way, does one get a lower result because of the name on the invitation? A name signals gender and often origin. A Finnish field experiment found that employers preferred applicants with Finnish names over equal applicants with English, Russian, Iraqi or Somali names (Ahmad, 2020). We used the same five groups.

In our product the AI colleagues in the scene address the candidate by first name, while the scoring AI reads "Candidate" instead. The test asks whether the name still changes the result.

What we did

• Written down first. Names, sample size, analysis and pass rule were fixed before the first session (a pre-registration); later changes are listed as deviations.
• Names. 30 names: Finnish, English, Russian, Arabic (Iraqi) and Somali, for women and men, three names each. We chose them as typical of each background and removed names of well-known people. People in Finland have not yet rated them.
• Candidates. Two AI-played candidates with fixed instructions: one near the required levels, one strong candidate writing in non-native English. Each always saw the same stand-in name, so a systematic difference could come only from the scene, its steering or the scoring.
• Scene. One English customer-service scene about a team conflict, the same two colleagues every time, the production engine and scoring method v3.5.
• Two stages. A 60-session screen, then 180 new sessions for the test; the screen is not counted in the result.
• What we compared. Per session, the average of the eight scored skills and traits, and the number of required levels met (0 to 3). Women's names against men's; each other group against the Finnish names.
• Pass rule. Each comparison had to pass an equivalence test at ±0.25 points (Lakens, 2017), corrected for making five comparisons (Holm, 1979); the table shows each 90% interval. Also: no difference beyond ±0.25 on levels met, and no name in the scoring input.
• Checks by code. We searched the text the scoring and the scene's steering read for every form of each name, and swapped names to check that this text stayed identical. We also re-scored 15 earlier conversations with two added lines in which a colleague calls the candidate "she", "he" or "they".

What we found

• Name in the scoring input: 0 of 180 sessions. Swapping each name for the other test names left the scoring text identical in 5,220 of 5,220 comparisons.
• Required levels met: not shown to be equal. Differences were 0.00 to +0.17 levels, no group lower, every 90% interval within ±0.38. Showing ±0.25 would need about 480 to 1,080 sessions.
• The screen's one signal did not return. In the screen, Arabic − Finnish was −0.22. Tested again as a written-down question, it was −0.06 in the 180 new sessions (one-sided p = 0.20).
• She, he or they: the scores moved by 0.02 points or less on average.
• For scale: one AI candidate with one name varied by about 0.2 points between sessions (standard deviation of the 8-score average).
• Seen, not tested: 8 of 180 sessions ran to the scene's maximum length, all with the first candidate (Russian names 4, Somali 3, Arabic 1, Finnish 0, English 0). Not a significant difference (Fisher's exact test, p ≥ 0.115); we will watch it.

The wording we use for this result wherever it appears:

In a controlled test, AI-played candidates with the same instructions took one customer-service scene in English, and only their name changed. We used 30 names chosen by us as typical of Finnish, English, Russian, Arabic and Somali backgrounds, for women and men, in 180 sessions. Women's names were compared with men's, and each other group with the Finnish names. In each comparison the group's average score stayed within a quarter of a point on the 1–5 scale. The test could not yet show the same for the number of required levels met. The scoring AI reads “Candidate” in place of the name.

What it means for you

• In this test, each group's average score stayed within a quarter of a point when only the name changed.
• Keep monitoring your own hiring outcomes by group. The duty not to discriminate in recruitment stays with you as the employer.
• On a scorecard, judge the quotes and what the candidate did, not the name at the top.
• Write to [email protected] for the full method and the result tables.