Deciding where AI belongs
Give your pilot a reason to stop
Define a pilot with a spending limit, evidence requirements, and explicit decisions to stop, repair, or expand.
Published · Collection dates are retrospective editorial dates.
A pilot should buy evidence for a decision. If nobody can say what decision comes at the end, it can become an indefinite subscription with a growing list of things to fix.
The label “pilot” does not make a deployment small. A trial connected to a live customer queue, broad permissions, and several business systems can create substantial operating responsibility. Give the trial boundaries before turning it on, including a reason to stop.
Stopping can mean the technology is unsuitable. It can also mean the trial answered its question: the workflow needs a clearer policy, another integration, or a different operating model. That is useful learning when the limits were agreed in advance.
Write the decision before the experiment
Start with the uncertainty. For a hypothetical service business: can a bounded assistant collect complete after-hours callback requests without making appointment promises, while leaving staff less cleanup than the current process?
That question identifies a task, an authority boundary, and an operating comparison. “Can we use AI?” does none of those things.
Then name the decision. You might expand to another request type, keep the current limited scope, repair and repeat a specific test, or stop using the system. Do not treat expansion as the default outcome that must be disproved.
Basecamp's scope-setting chapter is useful background for bounding a project by available time. Here, bound the pilot by both time and exposure: how much customer work and how much authority it receives.
Make the limit measurable
A time limit is helpful, but it is not sufficient. A trial could reach its spending limit early or encounter a serious failure on its first day. Define the relevant limits together.
| Limit | What to specify | Why it matters |
|---|---|---|
| Calendar | Start, review dates, and end date | Prevents an unattended trial becoming permanent |
| Spend | Setup, usage, staff review, and repair budget | Captures costs outside the subscription |
| Coverage | Channel, hours, location, and request types | Keeps the evidence tied to a known scope |
| Authority | Records it may read or change | Limits the consequence of a misunderstanding |
| Exposure | Which customers or requests enter the pilot | Supports staged, observable learning |
| Intervention | Who can pause it and how | Makes stopping operationally possible |
Fill in values from your business. There is no universal trial duration or acceptable error rate for every workflow. A request to quote opening hours and a request to modify an account deserve different controls.
Define what evidence counts
Inspect completed work, not just conversations. For intake, that might mean a usable record with the right queue and no unsupported promise. For a staff assistant, it might mean an approved response that required less total handling effort without losing accuracy.
Include the work transferred to people. A system that appears efficient because it sends every awkward case to staff may still be a poor investment. Conversely, a system that hands off a consequential request correctly should not fail the trial merely because it did not contain the conversation.
Collect a baseline with the same definitions. If the previous process counted every incoming message and the trial counts only eligible requests, the two completion rates are not comparable. Keep denominators visible.
NIST's AI Risk Management Framework supports a broader discipline of evaluation and risk management. Your local pilot sheet should translate that discipline into named evidence and decisions.
Choose stop conditions that do not need a committee
Some failures should trigger an immediate pause of the affected capability: unauthorized disclosure, an action beyond the agreed permission boundary, or a false commitment with a serious consequence. Write the response and owner before such a failure occurs.
Other findings call for a scheduled decision: review effort exceeds the budget, eligible volume is too low to justify the setup, or the integration cannot provide dependable completion evidence. These do not necessarily require an emergency shutdown, but they should prevent automatic expansion.
A fix also needs a boundary. “We need another week” is incomplete. Name the failure, the proposed change, the cost, and the test that will determine whether the change worked. Otherwise the repair stage becomes another open-ended pilot.
Hold a review that can end in no
Bring the operating owner, a frontline representative, the budget owner, and the person who can explain the technical evidence. Use actual examples alongside aggregate counts. Review clean completions, exceptions, rework, and the most consequential failure.
Separate what you observed from what you expect. “The system created these records correctly under this scope” is evidence. “It should save more once staff get used to it” is an assumption requiring a further test.
Use a short decision record:
- The question the pilot tested.
- The scope and operating conditions observed.
- Results against the original requirements.
- Costs, including human work.
- Remaining uncertainty and its consequence.
- The decision, owner, and next review date.
If you expand, name exactly what is expanding. Another location, another channel, and another action are different changes. Avoid treating success in one as permission for all three.
A well-run pilot leaves the business with a decision it can explain. Sometimes that decision is to keep a small, useful system. Sometimes it is to stop spending and fix the process first. The exit was part of the experiment from the beginning.
Sources
For help defining this workflow, explore our workflow integration work.
Keep reading
