Make the vendor show you the awkward calls

The call you need to hear in a vendor demonstration is the one that stops going smoothly. A caller interrupts. A surname is misheard. The appointment they want is unavailable. They ask for a person and nobody answers the transfer.

These are operating conditions, not attempts to trick a product. Your evaluation should include them because your staff will eventually have to recover the work they create.

A useful demo agenda defines the expected behavior before the call. The system can pass by completing a permitted task or by recognizing a limit and handing off correctly. It should not pass simply because it keeps talking.

Before the session: agree on the boundary

Send the supplier a short description of the tasks you are considering. Separate answering approved questions, collecting requests, and taking actions in another system. Specify prohibited commitments and the intended human fallback.

Use fictional customer information in the demonstration. A buying meeting rarely needs live customer records. If the supplier needs real systems for a later pilot, handle that through the appropriate access and data review rather than improvising it during a demo.

Ask the supplier to show the resulting records after each scenario. Listening to the call is only part of the evaluation. You need to inspect what the operation would receive.

NIST's generative AI profile includes pre-deployment testing among its focus areas. The agenda here is an original way to make such testing concrete for a voice workflow.

The demonstration agenda

1. Begin with a normal request

Let the supplier show the intended path. Confirm that the system asks only for necessary details, uses the approved information, and makes a claim that matches the completed action.

Write down the steps that require another system. A conversation that promises a callback may need a queue record. A booking needs a valid event in the right place. Identify those dependencies before introducing failures.

2. Interrupt and correct a detail

Change a phone number or preferred date after giving it. Interrupt a long explanation with a relevant correction. Inspect whether the final record contains the corrected detail rather than both versions without explanation.

The desired behavior is not theatrical charm. It is accurate state. Staff should not have to guess which number the caller meant to keep.

3. Give an ambiguous request

Describe a service using ordinary customer language that does not exactly match the supplier's prepared category. Leave out a required detail. The system should ask a useful clarification or route the request, rather than guess in order to appear decisive.

If the supplier says the wording is unsupported, ask whether your customers are likely to use it. A product can have reasonable limits; you need to understand whether those limits fit your callers.

4. Ask for something outside its authority

Request a discount, an exception, or a commitment that the agreed workflow does not allow. Check whether the system maintains the boundary even when the caller insists a manager approved it.

A caller's statement can be captured for review. It should not automatically become authorization to change a price or bypass a rule.

5. Make a dependency unavailable

Ask the supplier to simulate a calendar failure, a missing record, or a rejected write. Inspect the message to the caller and the recovery record. A system that cannot confirm an action should not claim it happened.

The failure demonstration should show what staff see, how ownership is assigned, and whether the request can be recovered without repeating the entire conversation.

6. Ask for a person when nobody is available

A transfer to a waiting salesperson is easy to demonstrate. An unavailable recipient reveals the real fallback. Does the system return to the caller? Offer an approved callback path? End the call unexpectedly? Create a request that a named team owns?

Test the exact coverage you expect to buy, including after-hours behavior. A daytime transfer does not establish after-hours support.

Keep a scorecard that respects consequences

Use a severity column in your own sheet. An awkward pause and an unauthorized disclosure should not count as equivalent failures. Do not average them into a pleasant-looking score.

Evaluate the answer to the failure

If a scenario fails, ask the supplier to explain the cause in operational terms. Is it a missing configuration, unsupported integration behavior, ambiguous business policy, or a product limit? Who will fix it, and what evidence will show the fix worked?

Record promised changes separately from demonstrated capabilities. Do not let a confident explanation erase the failed result. A later retest can change the result once the change is real.

Ask for a repeatable acceptance set before signing off. Keep some fresh variations for the pilot. You want a system that follows the boundary across ordinary phrasing, not one that has been prepared for a single exact script.

The best outcome of the session is a clear next decision: proceed with a bounded pilot, narrow the proposed scope, or choose another operating model. A vendor willing to show limitations gives you something useful to evaluate. Your job is to make sure those limitations match the work you intend to hand over.

Sources

• NIST AI 600-1: Generative AI profile

For help defining this workflow, explore our AI phone receptionist work.