Field Note 01 ยท 26 August 2026
First, what a certification run is, in one minute
Skip this if you have read any other entry on this site.
A certification run is a conversation, not a quiz. The learner talks to an AI character who plays the other side of a real situation: a customer who is angry, a direct report who is not listening, a senior colleague who keeps deflecting. The whole exchange is recorded as a transcript.
The transcript is what gets scored, and the score decides whether the person earns a credential. So the score has to do two things a fair examiner does. It has to give the same conversation the same mark every time. And it has to put a strong conversation above a weak one, every time, for reasons it can show.
Think of a driving test. The examiner does not watch you drive for forty minutes and then feel their way to a mark out of 100. They tick specific things at specific moments: checked the mirror before pulling out, signalled before the turn, held the car on the hill. The route was planned to contain those moments. That is the whole idea in this entry, and it took us a failure to find it.
The number wandered
A roleplay is the best way to see whether someone can actually do something. Give hard feedback. Calm an angry customer. Push back on a senior stakeholder. You cannot learn that from a quiz, and an attendance record will never tell you who has it. It is also the hardest thing we have ever had to score.
The first time we ran a conversation we knew was competent through our scorer, it came back at 50. We ran it again. It came back at a different number. Then we ran a weak conversation, and sometimes it landed on the same score as a strong one.
Nothing about that is usable when the score decides whether someone earns a credential.
The sensible idea, and why we built it
It is worth being clear about what the first scorer was, because it was not a careless design and the reason it failed is the lesson.
The idea was simple and it is the one almost everybody reaches for. Take the whole conversation. Hand it to an AI. Ask it to read the transcript the way an experienced manager would, and give a score out of 100 on each of the six behaviours the course was teaching. An expert human does roughly that when they watch a roleplay, so asking the machine to do the same seems like the natural first step.
And on any single conversation it looked as though it worked. You got a number back for each behaviour. The numbers sat in sensible bands. There was a paragraph of reasoning under each one that read well.
It collapsed the moment you asked the one question a credential depends on: would it give the same answer twice, and would it always put the better conversation first? It would not. The same transcript scored twice produced two different numbers, and a weak conversation could tie with a strong one.
The human consequence is the only one that matters. A learner could sit the same certification on two different days, perform identically, and pass on one and fail on the other. Nobody could tell them why, because there was no why. There was a paragraph of prose and a number.
Here is what did not fail. The AI was not stupid, and it was not reading the transcript wrong. We had asked it to do the one thing it is worst at: hold thirty exchanges in its head at once, weigh them all against each other, and squeeze that judgement into a single number. The question was the problem, not the answerer.
The part we are not going to pretend
The easy version of this story is that we built something naive and then got clever. That is not what happened, and the honest version is more useful.
The first scorer was carefully built. One model call scored every behaviour at once. It had defined score bands with named thresholds. It shipped with tests, and they passed.
It also carried a defect nobody had chosen. The band a learner landed in was supposed to come from the model's own answer. But when that answer did not match what the code expected, the code quietly worked out a band from the score instead. So the same question had two different answerers, and nobody reading a result could tell which one had produced it.
That was not an isolated lapse. An audit of our own system a few weeks later found eleven places where we asked an AI model for a judgement in prose and then parsed the prose, with nothing forcing the answer into a defined shape. The wandering score was the most visible symptom of a habit that ran right through the system. We fixed the habit first, then went back to the score.
The idea we borrowed
Medical schools solved a version of this forty years ago, and the shape of their solution is worth walking through slowly, because it is the shape of ours.
If you want to know whether a trainee doctor can handle a patient, you do not sit them in one long unstructured exam and give a gut-feel grade at the end. You send them through a series of short stations. At each one, a trained actor presents a deliberately scripted situation: the same chest pain, the same anxious parent, for every candidate. The examiner at that station does not score an overall impression. They score a short checklist of specific, observable things. Did they wash their hands. Did they ask about allergies. The result is the total across all the stations, and the pass mark is set by a defensible method rather than picked.
The medical world calls this an objective structured clinical examination, an OSCE. Four things in it map almost exactly onto our problem. The actor creates the difficult moment on purpose, so every candidate is tested on the same things. Small specific behaviours are scored, not an overall feeling. One station is noisy, but the set is reliable. And the pass mark is set on purpose rather than assumed.
It has a known weakness too, and it is worth stating. A checklist can reward ticking the boxes over doing the job, and a station is only as good as the script the actor was given. Both of those turned out to be true for us as well, and both are published on this site.
So the design became: an OSCE where the stations live inside one flowing conversation, and the actor and the examiner are both AI.
The move that fixed it
Stop thinking about a roleplay as a number of turns. Start thinking about it as a sequence of moments the conversation was built to contain.
We call each of those a beat. A beat is one engineered moment, designed so the learner gets a real chance to show one specific thing. The AI character is not just reacting with a personality. It is working an agenda. It will deflect. It will minimise. It will offer a vague close. Not because that is how the conversation happens to go, but because those are the moments the assessment needs to exist. It is the hill start on the driving route.
Then, at each beat, the AI answers a handful of narrow questions about that moment only. Did the learner name the issue plainly. Did they stay calm when the other person pushed back. The answer to each is yes, partly, or no. Yes is worth full marks, partly half, no nothing. Arithmetic does the weighing, and the pass line is a number set on purpose and written into the code.
The model never produces a score. That is the whole move. We had been asking it to do the thing it is worst at, and we started asking it to do the thing it is best at: answer one small, concrete question about a short passage of text.
There is a second thing this fixes, and it matters more than the reliability. The old scorer would happily mark someone down for composure in a conversation where nobody had ever pushed back on them. Under the new design, nothing gets scored that the conversation never gave the learner a chance to show. If there was no beat for it, there is no mark for it.
What the numbers said
We ran it as an experiment before we believed it.
The first test was small: one genuinely good conversation and one weak one, scored both ways, by four models from two different companies. Scored the old way, the cheapest model gave the good conversation and the weak conversation the identical mark. It could not tell them apart at all. Scored the new way, it separated them, and so did every other model. The difference came from exactly where you would want it to: the weak conversation had lost its composure under pressure, and a narrow question about staying calm caught that every time, where a mark out of 100 had smoothed it away.
Then we did it properly. Thirty six transcripts in one skill, written to span the range from strong to weak, rated blind by a human who did not know which was which, and scored by a jury of six AI models. Every one of the six agreed with that human well past the level the testing profession publishes as the bar for trusting an automated scorer in high stakes exams. And not one of them, on any transcript, ever ranked a weak conversation above a strong one.
What we learned that we did not expect
The wobble did not disappear. It became visible. Narrow questions at a fixed setting are stable. The few that moved between runs were the genuinely ambiguous ones, and now we can name them. A single wandering number tells you nothing about where it is wandering. A hundred small answers tell you precisely.
Disagreement turned into a signal. Scoring the same conversation with several independent models tells you something one score cannot. Where they agree, the verdict is safe to accept. Where they split, the split lands almost every time on a question of interpretation rather than a question of fact, which is exactly the kind of question a person should be looking at. That is the basis for sending the rest to a human.
One honest caveat on that. Certification today is scored by a single model. The jury is a capability we built and measured, not a safeguard currently running, and we would rather say so than let the word jury do work it is not doing.
Two rules fell straight out of the evidence, and we hold them. The pass line has to be set for each scoring model separately, because models differ in how harsh they are. And a single roleplay can never certify anyone. One station is noisy. Certification needs a set.
What this does not prove
The scale run is one skill and 36 transcripts, rated by one person. That person is our founder, and it is a limit we state everywhere these numbers appear. We have not measured how much two trained people agree with each other on the same conversations, so we cannot yet say how much of the remaining gap between a model and a human is the model's fault. That study is running.
The transcripts were written to a design rather than recorded from live learners, so they tell you how the scorers behave on material built to be scored. And agreement is not the same as accuracy in points: the jury tracks the human's ranking closely and still runs a few points below on the raw score. That gap has its own entry.
The part we keep
We publish the method and we publish the numbers. The part we do not publish is the library of checks itself, the worked examples that anchor them, and the rules for writing new ones. That is the actual work, and it is years of it. Knowing that small questions beat a single score will not build you a catalogue.
The examiner's clipboard
Go back to the driving examiner. Nobody would accept a licence decided by an examiner who watched the drive, thought about it, and wrote down 50. Not because the examiner is a bad judge, but because a number like that cannot be checked, cannot be appealed, and might come out as 61 tomorrow. What makes a driving test fair is the clipboard: a planned route, specific moments, a tick or a cross for each, added up in the open.
That is what we built. The machine holds the clipboard. It never holds the pen that writes the final mark.