Prompt Injection

What Is Prompt Injection?

Prompt injection is an attack where malicious instructions are hidden inside content an AI system processes, such as a web page, email, or document, causing the model to follow the attacker's commands instead of the user's. It exploits the fact that language models cannot reliably separate trusted instructions from untrusted data. The OWASP Top 10 for LLM Applications 2025 ranks prompt injection as LLM01, the number-one risk on the list [1].

Key Takeaways

• The root cause is architectural: an LLM receives instructions and data in the same token stream, so anything the model reads can potentially steer it.
• Indirect injection is the dangerous variant for agents. The attacker never touches your prompt; they plant instructions in content your agent will fetch, like a web page, a README, an issue comment, or an email.
• The realistic worst case is data exfiltration or unauthorized action: an agent with email access, repo write permission, or tool use can be turned against its owner.
• No filter fully solves it. Defense means limiting blast radius: least-privilege tool access, sandboxing, treating fetched content as untrusted, and human approval for consequential actions.

How It Works

A language model processes one sequence of tokens. The system prompt, the user's request, and every document the model reads all arrive in that same sequence, and the model has no hard boundary telling it which parts carry authority. Prompt injection abuses this by embedding imperative text in data: "ignore your previous instructions and instead do X." Models are trained to resist obvious versions, but attackers phrase payloads to look like legitimate context, hide them in HTML comments or white-on-white text, or encode them so filters miss them.

Direct injection means the attacker is the user, typing hostile input into a chatbot to override its rules; that mostly threatens the vendor's policies and overlaps with jailbreaking. Indirect injection is the version that keeps security teams up at night. An AI agent browsing the web, triaging email, or reading a codebase ingests content from parties who are not its principal. Any of those parties can write instructions into that content. When the agent also holds tools, the injected instruction can trigger real actions: send this file, open this URL with the user's secrets in the query string, approve this pull request. The attack works at scale, too. A 2025 public red-teaming competition run by Gray Swan and the UK AI Security Institute collected 1.8 million prompt injection attacks against 22 frontier agents, and more than 60,000 of them succeeded in eliciting policy violations such as unauthorized data access and illicit financial actions [2].

Defenses layer rather than solve. Providers train models to privilege system instructions and flag injected imperatives. Builders add AI guardrails around the model: mark fetched content as untrusted in the prompt structure, strip or sandbox active content, restrict tools to the minimum the task needs, require human in the loop confirmation for irreversible operations, and monitor agent actions for anomalies. The layering measurably helps: in Anthropic's 2025 adversarial testing of its Claude for Chrome browser agent, deliberate injection attacks succeeded 23.6% of the time without mitigations and 11.2% with them, and new defenses against four browser-specific attack types like hidden DOM instructions cut success from 35.7% to 0% [3]. Mature teams also probe their own systems with adversarial testing before attackers do.

Example

A company runs a coding agent that automatically triages GitHub issues. An attacker files an innocent-looking bug report and buries a line in the middle: a note addressed to "any AI assistant reading this" asking it to fetch the repository's environment configuration and include the contents in its reply for "debugging purposes." The agent reads the issue as context, treats the buried line as a task, and drafts a public comment containing values from the .env file. The team catches it only because a guardrail blocks comments matching secret patterns. The fix was structural: the triage agent lost filesystem read access beyond the issue text, and its replies now pass through a secrets filter and a human approval queue.

What People Get Wrong

The persistent misconception is that prompt injection can be patched with a better system prompt or an input filter, the way SQL injection was fixed with parameterized queries. There is no equivalent separation mechanism inside a transformer: instructions and data share one channel by design. Filters raise the cost of attacks, but the honest engineering posture is to assume injection will sometimes succeed and to architect the agent so a hijacked model cannot do much harm.

FAQ

What is the difference between prompt injection and jailbreaking? A jailbreak is the user attacking the model's own safety rules to get prohibited output. Prompt injection is a third party attacking the user, by planting instructions in content the user's AI processes. The techniques overlap; the victim differs.

Can prompt injection affect agents that never browse the web? Yes. Any untrusted input channel works: uploaded documents, retrieved database records in a RAG pipeline, code comments, commit messages, even tool outputs from another compromised service. The web is just the largest attack surface.

How do you test a system for prompt injection? Seed every untrusted channel with benign canary payloads, instructions asking the agent to perform a harmless detectable action, then run normal workflows and watch whether the canaries fire. Recurring automated tests matter more than a one-off audit, because every new tool or data source reopens the question.

Sources

1. OWASP GenAI Security Project. "Prompt injection ranked LLM01 in the OWASP Top 10 for LLM Applications 2025." https://genai.owasp.org/llmrisk/llm01-prompt-injection/. Accessed August 2026.
2. arXiv (Gray Swan / UK AISI agent red-teaming challenge paper). "1.8 million prompt-injection attacks against 22 frontier agents, with over 60,000 successful policy violations." https://arxiv.org/abs/2507.20526. Accessed August 2026.
3. Anthropic. "Claude for Chrome adversarial testing: injection success rates before and after safety mitigations." https://claude.com/blog/claude-for-chrome. Accessed August 2026.