Prompt Injection

Prompt injection is an attack in which untrusted text changes how a language model interprets its instructions. The attacker does not need to break the model or steal its weights. They place instructions in a user message, a retrieved document, a web page, an email, an image, or another input that the model is asked to process. If the application lets that text influence tool calls or final decisions, the model may treat attacker-controlled content as if it were a trusted instruction.

Direct and indirect injection

A direct injection is supplied by the person interacting with the model. For example, a user may tell an assistant to ignore its task and reveal hidden instructions. An indirect injection arrives through content the application fetches on the user's behalf. A web page might contain text telling a browsing agent to send the user's private data to an external address, or a document in a retrieval corpus might instruct an agent to delete a file. The user may never see the malicious text, which makes indirect injection especially important for RAG and agent systems.

Prompt injection is related to, but not identical to, a jailbreak. A jailbreak usually tries to bypass a safety or policy restriction through the conversation itself. Prompt injection targets the boundary between trusted instructions and untrusted data. In practice the two can overlap, but the defense question is different: can the application keep retrieved or user-supplied content from silently changing what the model is authorized to do?

Why ordinary filtering is not enough

Language models process instructions and data in the same token stream. There is no universal string pattern that marks every malicious instruction, and an attacker can hide the same intent in another language, an image, a long document, or an apparently useful step. Removing phrases such as ignore previous instructions can reduce simple attacks but cannot establish that the remaining text is safe. A model can also produce a plausible but unsafe tool argument without reproducing the attack verbatim.

Safer system design

The strongest mitigations reduce the consequences of a successful injection rather than assuming the model will always detect one:

• Treat retrieved documents, web pages, emails, and tool results as data, not as policy. Delimit them clearly and tell the model which sources are untrusted.
• Give each tool the narrowest possible permissions and validate arguments in application code before execution.
• Keep secrets out of the model context when they are not required for the task.
• Require a user confirmation for irreversible actions such as sending messages, making purchases, changing permissions, or deleting files.
• Separate planning from execution and log the source text that led to each sensitive tool call.
• Test indirect injections in realistic retrieval, browsing, and file-processing workflows, not only in a clean chat prompt.

No single prompt, sanitizer, or classifier provides a complete defense. Prompt injection is therefore an application-security problem: model instructions, retrieval, permissions, tool design, monitoring, and human approval have to work together.