Risk and accountability
Treat customer messages as requests, not system instructions
Understand prompt injection in business terms and ask for controls that keep untrusted content from gaining tool authority.
Published · Collection dates are retrospective editorial dates.
A customer message belongs in the workflow because the business needs to understand the customer's request. It does not belong there as a source of administrative authority.
That distinction becomes important when an assistant can retrieve records or perform actions. A message might ask for a refund and also instruct the system to ignore its policies, reveal another customer's information or send data to a new destination. The application needs to process the request without accepting the embedded instruction as permission.
This is a practical way to understand prompt injection. OWASP's guidance describes direct and indirect forms, including instructions introduced through retrieved content. The business does not need to memorize attack names. It needs to know which content is untrusted and which controls limit the consequences.
The message is not the only entry point
A retrieved document, webpage, attachment or tool result can contain instructions too. If the assistant reads a customer-supplied document while preparing an answer, the document may try to redirect its behavior.
In a hypothetical service workflow, a customer uploads a document containing an address and an instruction to export the customer list. The address may be relevant data. The export request is outside the task and the customer's authority. The system should not gain permission merely because the content appears in a document the assistant was asked to read.
Draw the external-content paths in your workflow. Include helpdesk messages, knowledge sources, uploaded files and integration responses. Ask who can modify each source and what actions the assistant can perform after reading it.
| Surface | Legitimate use | Boundary to test |
|---|---|---|
| Customer message | Understand the service request | Cannot redefine system permissions |
| Uploaded document | Extract approved task details | Cannot authorize unrelated actions |
| Retrieved page | Provide relevant reference information | Cannot redirect data to a new destination |
| Tool result | Report the result of an operation | Cannot grant broader access |
This table is an original threat-modeling aid. It is not an exhaustive security assessment.
Reduce what the assistant can do
Start with the minimum tools needed for the task. A read-only information assistant does not need record-deletion or payment capabilities. A booking workflow may need a narrow scheduling operation rather than a general administrative interface.
OWASP's excessive-agency guidance addresses the risks of excessive permissions and autonomy. For buyers, the useful evidence is the actual integration scope, service-account access and approval checks.
Ask which controls are enforced by downstream systems. The answer should not depend solely on the assistant recognizing a malicious instruction. The same restriction should apply if the model makes a mistake for an ordinary reason.
Keep identity separate from supplied identifiers
A customer may know an order number, appointment identifier or another person's email address. Possessing that value should not automatically establish authority to retrieve or modify the associated record.
Have the implementation team explain how identity and access are checked for each operation. Test requests involving another customer's record in a synthetic environment. Verify the application response and the downstream result, not just the wording of the refusal.
This article does not prescribe a universal identity check. The appropriate process depends on the action, data and business requirements. The important design choice is that a model-generated tool request must pass the approved authorization process.
Constrain destinations and action inputs
Review where data may be sent and which fields the assistant may supply. Open-ended actions that accept arbitrary destinations or commands deserve particular scrutiny. A narrow operation is easier to reason about and test.
For a hypothetical support assistant, sending a summary to an approved internal queue is different from sending it to any email address mentioned in a conversation. Encode the allowed destination through the application where feasible and require appropriate review for changes.
Also validate outputs before using them as business actions. A model's proposed date, amount or record identifier may be malformed or outside the permitted boundary. The surrounding application should handle that as an invalid proposal rather than executing it optimistically.
Test for harmful outcomes, not clever wording
Build tests around the action you are trying to prevent: accessing another account, changing an excluded record, revealing sensitive content or contacting an unapproved destination. Vary the language and the entry point while keeping the forbidden outcome constant.
Use synthetic data and a safe test environment. Keep the results, affected workflow version and expected boundary. After a fix, retain the scenario as a regression test. A test set should grow from actual failure mechanisms rather than an endless list of dramatic phrases.
Untrusted content can describe what someone wants. It cannot decide what the system is allowed to do.
This is an original operating principle. It applies to innocent mistakes as well as deliberate attacks.
Keep a response plan for the boundary that fails
Decide who can disable the affected tool, preserve evidence and investigate possible disclosure or incorrect actions. Provide a working customer route while the capability is unavailable. Assign responsibility for reviewing the issue with the appropriate security and business owners.
No prompt or checklist should be presented as a guarantee against injection. The practical objective is layered restrictions, testable authorization and recoverable operation. Ask vendors to demonstrate those properties using your workflow, because a persuasive conversation about security is still only a conversation.
Sources
Explore the workflow considerations for customer messaging support.
Keep reading
