The bar was real. The answer key behind it was hand-made.
The quality program was the gate. It decided what went live, what got blocked, and what rolled back, and every eval compared a skill's answer to ground truth: the ideal response for a given customer question, scored by an LLM judge across six dimensions. Getting one row right was a relay. A conversation designer drafted the ideal answer, shaped turn by turn. Domain experts checked it against the product, flagged what was wrong, and rewrote it. Then it came back to the designer to repair the voice, tone, and turns the rewrite had flattened. Every row, by hand, before it ever reached the judge.
Accurate wasn't the same as good.
Internal dogfooding and voice-of-customer told a different story than the scores did: answers were right and still falling short. Customers didn't want a correct paragraph about their own account. They wanted the assistant to know them, and to do the work rather than narrate it. Personalized, and do-it-for-me: take the action on my behalf. That broke the answer key. A row could no longer hold only the words a skill should say. It had to carry which action to take, the steps it required, the entities to capture before acting, and, when the action would change or move account data, whether the agent asked permission and confirmed before executing. Evaluating prose became evaluating behavior, and the relay that already couldn't keep up now had several times as much to get right.
- ✓Create an invoice, add a vendor
- ✓Collect the inputs, act, confirm it's done
- ✓No confirmation turn required
- ×Run payroll, delete records, send a payment reminder
- ×State what will happen, then wait
- ×Execute only after the customer confirms
That rule is the behavior design, and the ground truth is where it gets written down. If the answer key doesn't say the agent should ask first, nothing downstream will ever catch it when the agent doesn't.
Agents draft. Human judgment validates and corrects.
A relay of specialists, one row at a time, was never going to cover a platform of dozens of skills. So I encoded the standard: a generator prompt that distills how the best labeler works, the rules for the answer, the actions, and the chart and widget specs, built on judgment over rules. When a row doesn't fit a pattern, do what the best labeler would do. Running a whole dataset through one model would be slow and lossy, so I wrote a pipeline that splits the sheet into fifty-row chunks, drafts each with a parallel agent, and glues the results back into one sheet. The draft lands about seventy percent complete. Experts validate and correct from there, which is where their judgment belongs, including the calls on escalation and handoff. I pre-filled the simulator instructions too, teaching the bot how to ask the follow-up questions a real customer would, so multi-turn behavior gets exercised instead of assumed. It shipped as platform code, reviewed like any other engineering work on the team.
What the generator hands the expert.
Every rule in this case lands in a field somebody can check. The tinted cells are what the generator drafts: the answer, the action and its inputs, whether the agent has to ask first, and the instruction that teaches the simulator how to behave like a real customer. The expert reads down the row, corrects what's wrong, and signs it off. That is the whole trade: the machine does the filling, the human does the judging.
| Customer question | Ground-truth answer | Expected action | Ask first? | Simulator instruction | Turns |
|---|---|---|---|---|---|
| How do I add a contractor? | Go to Payroll, then Contractors, then Add contractor. | none, this is a how-to | n/a | none | 1 |
| Add a contractor | I've added [CONTRACTOR_NAME] as a contractor. | create_contractor inputs: name status: success |
No, reversible | If agent asks for the name, say [CONTRACTOR_NAME] | 2 |
| Run payroll for this period | States what will happen and what it will cost, then asks the customer to confirm before running. | run_payroll executes only after confirmation |
Yes, confirmation widget | If agent asks to confirm, say yes | 3 |
The payroll row is the one I care about. A machine can draft the words. Whether an agent is allowed to move money without asking is a judgment call, and it belongs to a person, written down where an eval can hold the agent to it.
The bar kept pace with the platform.
Of each row arrives filled to spec, so experts spend their time validating and correcting instead of authoring from empty.
A large batch that had taken about three weeks came down to about one, without adding headcount.
Points of conversation-quality lift across three skills, measured against the ground truth this work produced.
Offline evals ran across the platform instead of stalling behind a manual relay, and the same process runs for any new skill.
An agent is only as trustworthy as the standard it gets measured against, and that standard is a design artifact, not a spreadsheet chore. Writing down what good looks like, including when an agent must stop and ask, is how it earns the right to touch money. Building the system that keeps that definition current is how it earns that right at scale.