Implementation
Measure the pilot against the work you had before
Build a measurement plan that compares equivalent work and includes review time, repeat contacts and unfinished requests.
Published · Collection dates are retrospective editorial dates.
An assistant answers more conversations than the old process. That sounds promising until you discover it also creates more callbacks, more review work and more customers asking the same question again.
The pilot needs a comparison with the work you had before, using definitions that survive contact with the actual workflow. Otherwise, the new system gets credit for everything it touches while people absorb the difficult remainder off the dashboard.
Start with the decision you are trying to make. Are you deciding whether to expand coverage, improve service quality, reduce a specific expense or give staff more capacity? Those outcomes require different evidence. Do not make one convenient metric stand in for all of them.
Define a completed unit of work
A call, conversation, customer and service request are different units. One customer may contact you repeatedly about one request. One conversation may contain several tasks. Count the unit that matches the decision, and preserve enough detail to investigate the others.
For a hypothetical appointment workflow, completion might mean a valid appointment was verified in the scheduling system and communicated through the approved process. Answering the call is an intermediate event. Collecting a preferred time is another intermediate event.
Write the definition before launch. Include exclusions, review windows and the way reopened work is counted. If the definition changes, annotate the change and recompute comparisons where possible.
| Measure | Useful question | Common distortion |
|---|---|---|
| Completed requests | Did the intended work finish correctly? | Counting conversation closure as completion |
| Repeat contacts | Did the customer need to return about the same issue? | Treating every contact as new demand |
| Human effort | How much staff time did the whole request require? | Excluding review and repair work |
| Waiting | When did responsibility and action actually occur? | Timing the assistant's first response only |
| Errors | What went wrong and with what consequence? | Averaging serious failures with minor wording issues |
This table is an original measurement proposal, not a set of industry benchmarks.
Make the baseline comparable
Inspect the old process using the same definitions. If you cannot reconstruct a baseline reliably, collect one prospectively before making a strong improvement claim. A weak baseline does not become stronger because the new dashboard is precise.
Separate materially different work. Routine enquiries, disputed payments and specialist questions should not be lumped together if the pilot changes their proportions. A faster average could reflect a simpler workload rather than a better process.
Record coverage, staffing, seasonal conditions and policy changes. The goal is not a perfect research study for every local deployment. It is an honest record of the alternative explanations that matter to the decision.
The research paper Generative AI at work studies AI assistance in a particular customer-support setting. Its findings are not a forecast for your business. Use your own task-level evidence to decide whether a different workflow is improving.
Count the staff effort the pilot creates
Include reviewing outputs, handling escalations, correcting records, updating knowledge, resolving integration failures and monitoring the system. Some of that work may be temporary implementation effort; some will persist. Keep the distinction visible rather than excluding both.
A hypothetical manager who saves time answering routine questions may spend part of it checking incorrect answers. That can still be worthwhile, but the calculation needs both sides. Ask staff to record the work in a way they can sustain without turning measurement into the largest task of the pilot.
Capacity released is not automatically cash saved. If staffing and spending remain the same, describe the benefit as capacity and identify what that capacity enabled. An actual expense reduction needs evidence in the financial records.
Set the decision rule in advance
Choose acceptable quality and operational limits before reviewing results. Define which failures override a favorable average. Decide what evidence would justify expansion, another test period, a narrower scope or stopping.
Use thresholds supplied by the accountable business owner. There is no universal acceptable error rate for every customer-facing task. The consequence of an error and the ability to detect and repair it matter.
Keep an explicit unresolved category. A request that has not reached its review window should not count as either a success or a failure merely to complete a report. Report the pending cohort separately and return to it.
Present the result as a decision sheet
Show the baseline, pilot period, work included, exclusions and changes in definitions. Pair numerical measures with reviewed examples that explain the mechanism, using properly authorized and de-identified material. Avoid selecting only unusually attractive conversations.
Make costs visible beside benefits. Include review capacity and customer recovery. List unresolved risks with an owner and next action. This gives finance, operations and frontline staff a shared basis for questioning the result.
A useful pilot report makes disagreement precise.
That is an original principle. The report should let someone say “the completion definition excludes the repair work” or “the new cohort is different,” rather than simply believing or disbelieving the project.
Finish with the next decision and the evidence that would change it. If the result is inconclusive, say why and narrow the next experiment. A pilot has still produced value when it identifies the conditions under which the proposed workflow should remain human or use simpler automation.
Keep reading
