In short
- Before a score is given, code looks for every quote that supports it in the candidate's own replies. In the main scoring path, a quote that is not found does not count as evidence. In our latest test run, the scoring AI picked out 1,687 moments and code found 1,686 of them in the candidate's replies; the one it did not find was set aside.
- The assessor report and the candidate's feedback note are checked as well. Report quotations not found in the conversation fell from 1% to 0%. Praise in the feedback note that the scorecard did not back fell from 28% in the first version to 3%, then to 0%.
- These counts come from AI-played test candidates and small samples. A quote that is found proves the words were said. It does not prove the score is right.
The question
When a scorecard says the candidate wrote "I'll call the customer back before noon", did they? Language models can write fluent text that is not in their source (Ji et al., 2023). An invented quote is worse than none, because it looks like proof.
What we did
What the product does on every assessment.
- Quotes first, then scores. First, the scoring AI picks out the moments that matter, with the candidate's exact words. Then the server, using ordinary code and no AI, looks for each quote in the candidate's own replies. A quote that is found is kept, with its turn corrected if needed; one that is not found is not used.
- No checked quote, no number. The scoring AI reads the whole conversation, but a score must point to at least one checked moment as its evidence. Without one, the scorecard says "not enough evidence" instead of a number. One exception: a low score (1 or 2) may rest on a clear opening the scene gave — a request, someone upset or blocked, a conflict — that the candidate did not act on. Code checks that the scene said those words and that no checked moment for that skill followed; the scoring AI must then accept it. The scorecard shows it as a "Missed opening" with its turn. Such a score has no quote from the candidate.
- A top score needs two moments. A 5 needs at least two separate checked moments.
- Reports and feedback. Every quotation in the assessor report must be the candidate's exact words. In the feedback note, praise must point to the candidate's checked words from a moment the scoring counted as a clear strength, on a skill scored in the middle of the scale or above (when no skill reached that level, one small genuine moment). The server checks both and sends an answer with a problem back once to be corrected.
- You can look. Each quote on the scorecard shows its turn, and the recruiter can open the conversation at that turn.
How we measured it. We used AI-played candidates only, never real candidate data.
- Scoring: our test set of 30 conversations, each scored three times (90 scorings), and a separate set of 6 conversations, each scored three times (18 scorings).
- Reports: 12 scored conversations, two reports each, written by the first version and by the current version.
- Feedback notes: the same 12 scorecards, two notes each, for three versions of the note.
A report quotation counted as "not found" when a quoted phrase of three or more words appeared nowhere in the conversation. Praise counted as "not backed" when a quote praised in the note was not a moment the scoring had counted as a clear strength. Both checks were done by code.
What we found
| Check | Before | Now |
|---|---|---|
| Moments the scoring AI picked out that were found in the candidate's replies (test set, 90 scorings) | – | 1,686 of 1,687 (the one not found was set aside) |
| Quotes under scores that are the candidate's words (test set / separate set; by design, only checked moments can be cited) | – | 3,441 of 3,441 / 658 of 658 |
| Scorings that needed the back-up path | – | 0 of 108 |
| Report quotations not found in the conversation | 2 of 313 (1%) | 0 of 111 (0%) |
| Scored skills and traits whose report line names a turn or a quote | 94% | 100% |
| Feedback praise not backed by the scorecard, all 12 | 23 of 82 (28%), then 2 of 61 (3%) | 0 of 45 (0%) |
| Same, the 9 candidates who scored low | 38%, then 6% | 0% |
The "before" columns are earlier versions of the report and the note, tested on the same scorecards. The "now" figures come from the current versions (the report since 27 October, the feedback note since 28 October 2026). The report and feedback checks used 12 scorecards from the previous scoring method (v3.4); the checks do not depend on the scoring method.
What it means for you
- Every quote under a score is the candidate's own words at the turn shown, unless the scorecard marks it "unverified".
- In the main scoring path, a skill without checked evidence says so instead of showing a number. A low score may instead rest on a "Missed opening"; read that turn.
- Read the turn around a quote before you act on it. The scorecard supports your decision; a person decides.
- Praise in your candidates' feedback must point to what their scorecard backs; the server checks this and sends back an answer that does not.
Sources
- Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023). Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12), 1–38. https://doi.org/10.1145/3571730
- hire.center (2026). How we measure the judge: Results for scoring method v3.5, the assessor report and the feedback note. Internal methodology document.
- hire.center (2026). Evidence architecture: The two-stage judge and the narrative reporter. Internal methodology document.