Scoring behaviour with evidence: anchors, rater training and the quote behind every score

The problem with a number

Ask a manager to rate a candidate's "communication" from 1 to 5 and you get a number that mixes three things: what the candidate did, what the manager thinks "communication" means, and how the manager felt that day. Rating research has a name for the best-known distortion: halo, named in 1920. A rater who likes one thing about a person (confidence, fluency, a familiar background) rates everything else higher too. The ratings for different skills then move together, and the scorecard tells you about the overall impression, not the skills.

Three practices reduce this. None of them is new; all of them are regularly skipped.

1. Anchor every score level in behaviour

A behaviourally anchored rating scale (BARS) replaces "1 = poor, 5 = excellent" with a short description of what each level looks like in action. For "staying calm with an upset customer", a 2 might read "answers the complaint with a defence of the company", a 4 "names the customer's problem back to them before proposing a step". The idea was published by Smith and Kendall in 1963 and has been refined since.

Anchors are meant to do two jobs. They help raters agree, because they are matching behaviour to a description instead of to a feeling. And they make a score explainable: a 4 is a 4 because the candidate did what the 4 says. Write anchors for each skill separately, in the words of the job, and keep them short enough to hold in mind.

2. Train raters on a shared frame of reference

Even with anchors, raters bring their own standard. Frame-of-reference training fixes this: raters score the same sample cases, compare their scores with a reference, discuss the differences and recalibrate. It was proposed as an alternative to training that only warns raters about common errors, whose results were inconsistent, and meta-analyses in 1994 and 2012 found that it improves rating accuracy. For a small team, a short session with a few practice cases is a practical start.

The practical sign that training worked: two raters scoring the same answer land within one point of each other, most of the time. If they do not, the anchors are unclear, or the raters have not calibrated.

3. Point every score at evidence

The third practice is the simplest and the most often skipped: write down the specific thing the candidate said or did that earned the score. A score with a quote next to it can be checked by a colleague, explained to the candidate, and defended later. A score without one cannot.

This also changes how raters work. Looking for the quote forces attention onto what happened instead of the overall impression, which is exactly what halo feeds on. And it makes "not enough evidence" a legitimate outcome: if the candidate never had an occasion to show a skill, the honest score is none, not a 3.

What this means for a structured interview

• Same questions for everyone, chosen to give each skill an occasion to appear.
• One written anchor set per skill, shared by all interviewers.
• Each interviewer notes the behaviour behind each score during the interview, not after.
• Interviewers score independently first, then compare. Large gaps go back to the notes.
• Keep the notes. They are the record of why.

When the rater is an AI

AI scoring of open answers has arrived in hiring. The same research applies, with two additions.

First, the known biases are documented. Studies of large language models used as judges find that they can favour longer answers, often prefer the first of two options, and can be steered by instructions hidden in the text they score. A 2024 study in Nature found language models judging people more negatively when the same meaning was written in a dialect (African American English) rather than in standard American English. None of this is a reason to reject AI scoring; it is the list of things to test for and guard against.

Second, the three practices can be checked at scale. An AI rater can be given the same written anchors as human raters; its scores can be compared with those of trained human raters on the same cases; and it can be asked to cite the exact words behind each score, which a simple program can check against the transcript. A quote that does not appear in the transcript is a warning sign, and a score that rests on it should not be used.

Questions to ask any vendor:

• Are the score levels defined by written behavioural anchors? Can we read them?
• How well does the scorer agree with trained human raters on the same cases, and how stable is it when it scores the same case twice?
• Does every score point to the candidate's own words, and is that quote verified?
• Has the scorer been tested for the known biases: answer length, non-native language, names, hidden instructions?
• What happens when there is not enough evidence for a skill?

Candidate reactions

Research on how candidates react to selection methods finds that they view a procedure more favourably when it looks related to the job, and that they rate interviews and work samples more favourably than personality questionnaires. Candidates with positive views also think better of the employer. A job-like task scored with visible evidence is built to look job-related, and it gives the candidate something a number cannot: a reason.