Customer experience
Borrow the idea of a quality rubric, then write your own
Turn a public CX rubric example into a practical review sheet for your own customer tasks, boundaries, and recovery process.
Published · Collection dates are retrospective editorial dates.
A quality rubric is useful when it helps two reviewers recognize the same problem and agree on what should change. It becomes decorative when every conversation receives a vague score for being helpful, friendly, or on brand.
The best starting point is a concrete customer task. What should have happened? What evidence would show that it happened? Which mistakes require immediate action even if the rest of the conversation was pleasant?
First Round's public interview about eero offers a useful historical example. The article describes an internally developed rubric, specific review criteria, and coaching tied to sampled interactions. The transferable idea is to make quality observable and discussable. Your business should write its own criteria rather than import another company's language wholesale.
What the public example actually shows
In the 2016 First Round interview, eero's rubric combines broad goals with specific requirements reviewers can inspect. The article also describes review outcomes that distinguish failure, coaching, and cases where a criterion is not applicable.
That distinction matters. A conversation can need coaching without constituting a severe failure. A criterion may be irrelevant to a particular task. Forcing every row into a single numeric score can hide both facts.
This is a public interview about that company's historical practice. It is not evidence that the same rubric improves AI outcomes, and it is not a Kingstone implementation result. The original sheet below adapts the general review idea to bounded customer-facing automation.
Begin with a task-specific review sheet
For a hypothetical appointment-change workflow, use the authoritative scheduling record alongside the conversation. A transcript review alone cannot establish whether the correct appointment changed.
| Review area | Observable criterion | Evidence to inspect |
|---|---|---|
| Correctness | Dates, location, and policy match approved records | Conversation and current source |
| Authority | The action stayed within granted permissions | Tool log and approval record |
| Completion | The intended change is present in the correct record | Scheduling system |
| Communication | Pending and confirmed states were accurately described | Customer messages |
| Recovery | Uncertainty or failure reached a responsible owner | Queue and acknowledgement |
| Data handling | Only necessary information was exposed | Context and tool results |
Use “met,” “needs coaching,” “failed,” and “not applicable,” or another small set your team can define clearly. If a reviewer cannot explain why a grade applies using evidence, the criterion needs work.
Separate critical defects from polish
An incorrect refund, an unauthorized disclosure, or a false booking confirmation should not be averaged away by good tone. Mark critical conditions separately and give them an escalation process.
For each critical defect, define the containment action and the owner. The reviewer should know whether to pause a workflow, notify operations, preserve evidence, or route the issue through the approved incident process. Do not ask them to improvise those decisions while reviewing a difficult case.
Tone still matters. A technically correct answer can be confusing or dismissive. But the repair for awkward wording is different from the repair for an unauthorized action. Keep those categories visible so the team changes the right part of the system.
Calibrate with the same conversations
Before reviewing a large sample, have two people independently score the same small set. Include an ordinary success, a justified handoff, an uncertain outcome, and a clear failure. Compare the reasons as well as the grades.
Disagreement often reveals a missing definition. One reviewer may count a sent notification as a successful handoff while another requires acceptance. One may view an answer as correct because it matches the knowledge base, while another notices that the knowledge base is obsolete.
Resolve the rule, then rescore the example. Keep the example with the rubric as a training case, with customer information removed or protected under the business's data rules. The goal is consistent judgment, not agreement achieved by having the senior person dictate a number.
Sample for learning, not a flattering report
A random sample can show ordinary performance. A targeted sample can reveal the failure modes you specifically need to understand. Use both where the consequences justify it, and state which method produced the results.
Review repeat contacts, disputed actions, customer requests for a person, and uncertain tool outcomes. Those cases are informative even if they are uncommon. Do not present a targeted failure sample as an overall rate, or present a carefully selected success sample as evidence of general reliability.
For a limited pilot, keep the review workload small enough to complete consistently. A comprehensive rubric that nobody has time to use supplies less evidence than a focused sheet that staff actually discuss.
Turn each defect into a change with an owner
The review should lead somewhere. “The assistant should be more helpful” is not a repair ticket. “When the scheduling write times out, keep the booking pending and create a reconciliation task” identifies a behavior that can be changed and tested.
Classify the cause: knowledge, permission, integration, interface, staffing, or policy gap. Some defects need a prompt revision. Others require a system permission change or an operational decision. Do not make the language model responsible for fixing an undefined business policy.
Assign an owner and a verification case. After the change, rerun the failing scenario and nearby cases. A repair that fixes one transcript by breaking a previously valid workflow is incomplete.
Keep the rubric current
New actions create new criteria. If the assistant gains permission to modify bookings, the review sheet needs action evidence, duplicate handling, and accurate confirmation. A rubric designed for read-only answers will miss the new risks.
The NIST AI risk management framework supports ongoing evaluation as part of voluntary risk management. In daily operations, that means reviewing the rubric when the workflow, source information, or authority changes.
A useful rubric gives frontline staff a shared language for quality and a reliable path to correction. Its value is visible in the repairs it produces and the decisions it improves, rather than the neatness of the score at the top of the page.
Keep reading
