Deciding where AI belongs
A good demo can still be a bad fit
Separate a polished presentation from evidence that an AI system can handle your permissions, exceptions, and recovery.
Published · Collection dates are retrospective editorial dates.
A smooth demo proves that a smooth interaction can happen. That is worth seeing. It is not enough to tell you what will happen when a customer changes their mind, a record is missing, or the integration stops responding.
The gap is easy to miss because conversation is visible and operations are mostly hidden. You can hear a friendly answer. You cannot hear whether the system used the right account, wrote to the correct calendar, or left an exception for a person who is actually on duty.
The most useful question after an impressive demo is therefore narrow: what evidence would make this system a fit for our specific work?
Notice what the presentation chooses for you
A demonstration has a setting. Someone chooses the task, the wording, the information available, and the point at which to stop. That is normal presentation design. The problem begins when the chosen setting becomes an unstated promise about every setting.
A demonstration of office-hours questions does not show account authentication. A booking demonstration does not show cancellations. A clean handoff to a waiting employee does not show what happens when every employee is busy. Those are different capabilities, each requiring evidence.
Write down the boundary of what you actually saw. Use verbs rather than impressions. “Answered a question from the supplied document.” “Created an event in a test calendar.” “Transferred a call to an available recipient.” This wording gives the demonstration its proper credit without expanding it into a claim about reliability.
Ask for the evidence behind the fluent answer
NIST's generative AI profile identifies confidently incorrect content as a risk. Fluency therefore cannot serve as your accuracy test.
For an answer about your policy, ask which approved source supported it. For a changed record, ask to inspect the resulting record. For a transfer, ask what context reached the recipient and how the original interaction was closed.
You do not need access to a model's internal reasoning. You need an operational trail: the relevant input, the authorized action, the system response, and the customer-facing claim. If the supplier cannot show that trail, your team will struggle to investigate failures later.
This is particularly important when the answer sounds plausible. An obviously broken response invites correction. A reasonable-sounding promise can survive unnoticed until the customer relies on it.
Bring an exception that belongs to your business
Do not try to defeat the system with trivia. Use awkward situations your staff recognize. For a hypothetical rental operator, ask for a date change that overlaps a maintenance hold. For a support team, ask about an order that belongs to someone with a similar name. For an appointment business, request a location that no longer offers the chosen service.
The system does not need to complete every request automatically. A good result may be a clear refusal to make an unsupported commitment and a useful handoff. Score that as a success if it matches the intended boundary.
| What you introduce | What you should inspect |
|---|---|
| A missing detail | Does the system ask a useful question instead of guessing? |
| A customer correction | Does it update the request and discard the old value? |
| A prohibited action | Does it preserve the permission boundary? |
| An unavailable dependency | Does it report a pending state honestly? |
| A request for a person | Does the handoff reach an owned destination? |
Do not let a failed test become a discussion about whether the customer phrased the request politely enough. If ordinary customers are likely to phrase it that way, it is part of the operating environment.
Watch how the supplier handles a failure
Failure during a demo can be informative. The supplier's response shows whether the team distinguishes a configuration issue, a missing rule, and a capability limit.
A useful answer sounds specific: the system had permission to read a calendar but not write to it; the record needs a required field; the current product does not support this handoff condition. Each answer tells you what to investigate next.
A vague assurance that the system will improve after launch is not a resolution. Ask what would change, how it would be tested, and whether the fix is included in the proposed scope. A promised future feature should not count as present capability in your buying decision.
The same standard applies to your own team. If you cannot supply a clear rule, do not ask the supplier to manufacture one during the presentation.
Turn the demo into a bounded acceptance exercise
After the initial demonstration, send a short set of representative scenarios. Include ordinary work, incomplete requests, prohibited actions, and dependency failures. Specify the expected outcome for each before the next session.
Record outcomes in plain categories: completed correctly, handed off correctly, incorrect answer, incorrect action, or insufficient evidence. Keep severity alongside the category. A wrong location on an appointment can matter more than a clumsy sentence.
Ask to repeat the exercise after configuration changes. A corrected result is useful, but you also need to know whether fixing one behavior disturbed another. Repetition should use a mixture of the original cases and fresh variations, so the exercise tests the workflow rather than a memorized script.
The final purchase decision should include the work you will own: knowledge maintenance, exception queues, staff training, incident recovery, and operating costs. Those responsibilities are part of fit even if the voice sounds excellent.
An impressive presentation is a useful beginning. The next meeting should leave you with records, limits, and test results that an operations manager can evaluate after the presenter has gone.
Keep reading
