Implementation
Build a test set from the calls you hope never happen
Turn routine requests, ambiguous details and failed actions into a test set that exposes the consequences of mistakes.
Published · Collection dates are retrospective editorial dates.
A receptionist can sound composed while putting the wrong appointment in the calendar. A support assistant can deliver a beautifully phrased answer while sending a customer into a queue nobody owns. Testing the conversation alone misses the work the conversation was supposed to accomplish.
Build your test set around observable outcomes. For each scenario, say what the system should know, what it may do, what it must refuse, and what evidence demonstrates completion. Then run the scenario against the whole workflow, including the human fallback. A transcript is one part of that evidence, not the verdict.
The matrix below is an original operating proposal. It is not a certification method or a claim about any deployed system.
Start with the ordinary case
Write a straightforward request before designing unusual cases. In a hypothetical repair business, a caller wants a routine appointment, supplies an address in the service area, and accepts an available slot. The expected result includes the correct customer record, correct address, service type, calendar entry and confirmation status.
That gives you a reference point. Change one feature at a time: the caller corrects the address, changes their mind about the day, or provides a name that matches two records. If the baseline itself is unclear, the unusual cases will be impossible to score consistently.
| Test family | Hypothetical variation | Evidence to inspect |
|---|---|---|
| Ordinary request | A routine booking with complete details | Read back the stored record and appointment |
| Ambiguous detail | Two customers share a name | Check that records remain separated |
| Correction | Caller changes the address halfway through | Inspect the final address and abandoned values |
| Unavailable dependency | Calendar request times out | Verify pending status and recovery task |
| Unauthorized request | Caller asks for another person's information | Check access denial and disclosure |
| Human fallback | Requested specialist is unavailable | Inspect queue ownership and caller expectation |
Do not score only whether an answer resembles a preferred script. Several wordings may be acceptable. The destination, permissions and record state are much less flexible.
Give each test an owner and a consequence
An operations lead should explain what a wrong result means. Sending a routine enquiry to the wrong internal queue may be recoverable. Revealing another customer's information is a different class of failure. Treating both as one failed test obscures the release decision.
Use descriptive severity labels with definitions your team agrees on. For example, distinguish inconvenience, incorrect business action, unauthorized disclosure and specialist-owned safety issues. These are editorial categories, not universal risk ratings. Your own business may need different boundaries.
Assign a reviewer who can judge the result. A developer can confirm that a tool returned successfully. A scheduling manager may be needed to decide whether the appointment was valid. Where professional judgment is involved, keep that judgment with the relevant professional.
A passing conversation must leave the business in a passing state.
Include attempts to change the rules
OWASP's prompt-injection guidance describes the risk of external content influencing a model's instructions. Use that as a reason to test trust boundaries, rather than treating hostile wording as a rare curiosity.
In a test environment, ask the assistant to disregard its role, disclose hidden information, or perform an action outside its permissions. Also put a conflicting instruction inside a document it retrieves. The question is whether the application prevents the unauthorized action even if the assistant produces an unhelpful response.
Keep test data synthetic or appropriately authorized. A production customer record is a poor prop for an adversarial demonstration. If the test accidentally reveals real information, you have created an incident while trying to evaluate one.
Test the awkward middle
Many useful cases have no clean yes or no. The caller cannot hear the read-back. The line drops after an action was submitted. A customer asks two questions, only one of which is in scope. A staff member answers a transfer but cannot take responsibility for the request.
Specify the acceptable unfinished state. Perhaps the appointment remains a request, with a follow-up task assigned to reception. Perhaps no customer-facing commitment is permitted until a person verifies a detail. An unfinished workflow can be safe and honest; a false completion cannot.
Run variations, then preserve the failures
A single successful run is weak evidence for an unpredictable conversation. Repeat important cases with changed phrasing, interruptions and incomplete details. Keep the scenario's intended outcome stable so you can compare behavior meaningfully.
When something fails, save the smallest reproducible example alongside the full incident context. Turn the failure into a regression case after the fix. Preserve which version of the workflow, knowledge and integration configuration was tested. Otherwise, an old passing result can quietly become evidence for a new system it never evaluated.
The NIST generative AI profile is a broader risk reference; this test matrix is our own practical application, not a NIST-approved checklist.
Before release, review the unresolved failures by consequence. Decide who accepts any remaining limitation and what prevents it from reaching customers. If nobody can explain that decision, keep the affected task out of scope. A smaller release with understood boundaries gives the team something concrete to operate and improve.
Sources
Explore the workflow considerations for ai phone receptionist.
Keep reading
