A roleplay is the best way to see whether someone can actually do something. Give hard feedback. Calm an angry customer. Push back on a senior stakeholder. You cannot learn that from a quiz, and you cannot tell whether someone has it from their attendance record.
It is also the hardest thing we have ever had to score.
Our first design did the obvious thing. Take the whole conversation. Hand it to an AI. Ask for a score out of 100 across the six behaviours the course was teaching. Get a number back.
The number wandered.
A competent conversation would come back at 50. A weak conversation would sometimes land on the same score as a strong one. Run the same transcript through twice and the two answers disagreed. Nothing about that is usable when the score decides whether someone earns a credential.
The part we are not going to pretend
The easy version of this story is that we built something naive and then got clever. That is not what happened, and the honest version is more useful.
The first scorer was carefully built. One model call scored every behaviour at once. It had defined score bands with named thresholds. It shipped with 42 tests, and they passed.
It also carried a defect nobody had chosen. The band a learner landed in came from the model's own answer, and, when the model's answer did not match what the code expected, from a second calculation derived from the score. Two sources of truth for one question, reconciled quietly, with no way for anyone reading a result to know which one had produced it.
That was not an isolated lapse either. An audit of our own backend a few weeks later found eleven places where we called a model, and not one of them constrained what came back to a defined structure. We were asking models for judgement in prose and then parsing the prose. The wandering score was the most visible symptom of a habit that ran right through the system.
We fixed the habit first. Then we went back to the score.
The idea we borrowed
Medical schools solved a version of this forty years ago.
If you want to know whether a trainee doctor can actually handle a patient, you do not give them one long unstructured exam and a gut-feel grade. You run an OSCE. The candidate moves through a series of short stations. At each one, a trained actor presents a deliberately scripted situation: the same chest pain, the same anxious parent, for every candidate. The examiner scores that station against a short checklist of specific, observable things. Did they wash their hands. Did they ask about allergies. The candidate's result is the aggregate across all the stations, and the pass mark is set by a defensible statistical method rather than a guess.
Four things in that design map almost exactly onto our problem.
The actor creates the difficult moment on purpose, so every candidate is tested on the same things. You score small specific behaviours rather than an overall impression. One station is noisy and the set is reliable. And the pass mark is calibrated rather than assumed.
So: an OSCE, where the stations live inside a single flowing conversation, and the standardised patient and the examiner are both AI.
The move that fixed it
Stop thinking about a roleplay as a number of turns. Start thinking about it as a sequence of beats.
A beat is one engineered moment, built so the learner gets a real chance to show one specific thing. The AI character is not just reacting with a personality. It is working an agenda. It will deflect. It will minimise. It will offer a vague close. Not because that is how the conversation happens to go, but because those are the moments the assessment needs to exist.
Then, at each beat, the AI answers a handful of narrow questions about that moment only. Did the learner name the issue plainly. Did they hold the line without escalating. Yes, partly, or no. Short context, one question at a time, concrete examples of what a yes and a no look like.
And then arithmetic does the weighing. Yes is 1, partly is 0.5, no is 0. Averages roll up. The pass line is a calibrated number in code.
The model never produces a score. That is the whole move.
We had been asking the model to do the thing it is worst at, which is to hold many considerations in mind across a long transcript and compress them into one number. We started asking it to do the thing it is best at, which is to answer a small concrete question about a short passage of text.
There is a second thing this fixes, and it matters more than the reliability. The old scorer would happily mark someone down on composure in a conversation where nobody had ever pushed back on them. Under the new design, nothing gets scored that the conversation never gave the learner a chance to show. If there was no beat for it, there is no mark for it.
What the numbers said
We ran it as an experiment before we believed it.
Two transcripts, one genuinely good and one weak, scored both ways, across four models from two different vendors.
Scored holistically, the cheapest model gave the strong conversation and the weak conversation the identical mark. It could not tell them apart at all. Scored the new way, decomposed into small checks at engineered moments, it separated them correctly. So did every other model.
Then we did it properly, at scale. Thirty six transcripts in one skill, rated blind by a human who had no idea which had been designed to be strong and which to fail, and scored by a jury of six models. Every model agreed with that human well past the level the field publishes as the bar for qualifying an automated scorer in high stakes testing. Not one of them ever ranked a weak conversation above a strong one. And on three quarters of all the individual judgements, all six models agreed with each other, which is the share that needs no human review at all.
Pilot run. 4 models across 2 vendors, 2 transcripts, 200 scoring calls, 0 errors. Holistic scoring on the cheapest model returned an identical 65 for the strong and the weak transcript. Decomposed scoring separated them on every model. After one refinement to a single check that had been asking two things at once, the cheapest model tracked the human rating to within 2.8 points of average error, a result three models across two vendors agreed on.
Scale run. 36 transcripts in one skill, rated blind by one human rater, scored by a jury of six models.
- Agreement with the human: weighted kappa 0.87, ICC 0.96.
- Strong versus weak discrimination: AUC 1.000 on every model.
- Where all six models agreed, they matched the human 91 percent of the time. That was 76 percent of all judgements, which is the proportion that needs no human review at all.
ICC is the intraclass correlation coefficient: how much of the variation in scores is real difference between learners rather than disagreement between raters. Weighted kappa is agreement corrected for chance, counting a near miss as less wrong than a wide miss. Both run 0 to 1, and the published bar for qualifying an automated scorer in high stakes testing is 0.70 for both. AUC at 1.000 means no strong conversation was ever scored below a weak one.
The interval on that 0.96 is not exactly recoverable, because the raw scoring outputs from the run were lost. An approximation from the point estimate alone gives 0.92 to 0.98, and we label it approximate wherever we quote it rather than dropping the interval and publishing a bare number.
What we learned that we did not expect
The wobble did not disappear. It became visible. Narrow questions at a fixed setting are stable, and most of them returned the same answer on every re run. The ones that moved were the genuinely ambiguous ones, and now we know exactly which. A single wandering number tells you nothing about where it is wandering. A hundred small answers tell you precisely.
Disagreement turned into a feature. We score with several independent models. Where they agree, the verdict stands. Where they split, that specific item goes to a human. The split items cluster on the interpretive checks, which are exactly the ones a person should be looking at.
Two rules fell straight out of the evidence, and we hold them. The pass line has to be calibrated for each scoring model, because models differ in how harsh they are. And a single roleplay can never certify anyone. One station is noisy. Certification needs a set.
What this does not prove
The scale run is one skill and 36 transcripts, rated by one person. That person is our founder, and it is a limit we state in every place these numbers appear. What we have not yet measured is how much two trained people agree with each other on the same conversations, which means we cannot yet say how much of the remaining gap between a model and a human is the model. That study is running.
The transcripts were also written to a design rather than recorded from live learners. They tell you how the scorers behave on material built to be scored.
And agreement is not the same as accuracy in points. The jury tracks the human's ranking and judgement closely, and it still runs a few points below the human on the raw score. That gap is its own story, and it has its own entry.
The part we keep
We publish the method and we publish the numbers. The part we do not publish is the library of checks itself, the worked examples that anchor them, and the rules for writing new ones. That is the actual work, and it is years of it.
Knowing that decomposition beats holistic scoring will not build you a catalogue.
Since then
Dated 26 August 2026.
This method is now the foundation everything else stands on, which means its failures became our failures to find. It has had two significant ones since, and both are published here: our own checks turned out to reward describing a skill over doing it, which failed genuinely strong learners for months, and a coverage claim in our spec turned out never to have been built.
We also stopped scoring in a way that let one bad grader kill a result, and we scheduled a re run of the original six models on the original 36 transcripts to test whether the numbers have drifted. Some of those model versions have already been retired by their vendors, which is itself the finding.
For where the numbers stand today, see the validity study (forthcoming). For why the biggest model was the worst scorer, see method beats model size. For what happened when this instrument caught its own defect, see the narration trap.
Ask any vendor for their human rater agreement numbers. These are ours, with the n and the limits attached.