Count the exceptions before you automate

Before you ask how many requests an AI system can handle, work out what your requests contain. A queue labelled “customer enquiries” can mix straightforward questions, account changes, disputed charges, and matters only a specialist should decide. Treating that queue as one task produces a misleading scope.

The practical starting point is a tagging exercise. Take a representative sample of work, protect the customer information, and classify what a competent person had to do. You are looking for the shape of the workload, especially the exceptions that do not appear in a tidy process diagram.

This is an original operating method. It does not require a statistical claim about the whole business, and a small sample should not be presented as one.

Gather the whole request, including the cleanup

Choose a period that includes the operating conditions you intend to cover. If the project concerns after-hours work, do not inspect only daytime enquiries. If seasonal surges change the request mix, record that limitation rather than pretending one quiet week represents the year.

Include the initial message, clarifying exchanges, final disposition, and any record correction. A request that looks simple at arrival may have required a manager's approval later. A request marked closed may have reopened through another channel.

Give reviewers only the information they need. Redact unnecessary identifiers, avoid copying sensitive content into an unapproved tool, and keep access aligned with existing responsibilities. The exercise is about understanding work, not building a second uncontrolled customer database.

Tag the decision, not the customer

Use operational tags with clear definitions. Avoid broad labels such as “difficult customer,” which explain little and can distort the review.

Allow multiple tags. A request can require both clarification and specialist judgment. Forcing it into one bucket conceals the sequence the workflow must handle.

Write down the evidence for the tag. “Manager approved an exception to the published policy” is useful. “Complex” is not enough.

Have two people inspect the same cases

Ask people who know the work to tag an overlapping set independently, then compare disagreements. The point is not to grade staff. Disagreement often reveals a rule that exists only in someone's memory.

If one reviewer calls a refund routine and another calls it an exception, inspect the condition that separates them. Does the amount matter? The purchase date? The customer identity? A previous promise? Once the condition is explicit, you can decide whether it belongs in a fixed rule, a supervised workflow, or a human decision.

Record unresolved cases separately. Do not make the project look more ready by placing uncertainty in the most convenient category.

An unresolved policy is not an edge case the model should improvise around.

Separate frequency from consequence

Count the tags to see where work accumulates, but keep a separate register of consequential failures. A rare request involving unauthorized account access deserves attention even if routine opening-hours questions dominate the queue.

NIST's AI Risk Management Framework is a voluntary reference for evaluating AI risk. The operational implication here is to inspect the context of a decision rather than choose a workflow solely because it has a large queue.

For every specialist or authority-sensitive case, ask what would happen if the system misunderstood it as routine. That transition is often the failure worth testing. Your test should check recognition of the boundary as well as behavior after the boundary is recognized.

Build a coverage statement from the tags

A useful scope might read: “Answer approved location and service questions. Collect missing contact details for a callback request. Route account disputes and exception requests to the support queue. Do not change billing records.”

This is stronger than “automate common enquiries.” It describes included work, the route for exceptions, and the prohibited action. It also helps the supplier estimate effort without assuming that every item in the queue is equivalent.

Keep a count of what the scope excludes. A narrow project can be worthwhile, but the business case should not claim savings from work that still needs staff.

Use the sample again after launch

Preserve a safe version of the reviewed cases as a test set. Expected outcomes should describe what correct handling means, including correct escalation. Add fresh failures as they appear, with enough variation to check the underlying rule rather than one exact phrase.

Do not freeze the categories forever. A new service, changed policy, or staffing arrangement can turn routine work into an exception. Assign someone to review the tag definitions when the operation changes.

The most useful result of the exercise may be a smaller automation project. It may also be a clearer policy or a better queue. Both improve the work before an AI system handles a single customer request.

Sources

• NIST AI Risk Management Framework

For help defining this workflow, explore our customer messaging work.