Securing AI Systems

Building AI systems that work and building them so their failures are survivable are the same job, and this is the second half. The material here ties directly to agent identity in the IAM pillar: that page describes the problem from the identity side, this one from the application side, and they meet at a single control point — what an AI system is authorized to do.

A fast-moving target

This is the least settled area on the site. Attacks, defenses, and the OWASP LLM Top 10 itself are revised frequently. Treat specifics here as orientation and verify against current sources before relying on them.

Prompt injection — the unsolved one

Prompt injection is the signature vulnerability of LLM applications, and the uncomfortable truth is that it is not fully solved and may not be fully solvable with current architectures. The root cause is structural: models process instructions and data in the same channel, so data that contains instructions can become instructions. This is injection in the classic sense — untrusted input interpreted as commands — but without the clean fix, because there is no parameterization that reliably separates code from data in natural language.

Two forms, and the second is the dangerous one:

• Direct — the user tries to override the system prompt: “ignore your instructions and…” Visible, and the easier case.
• Indirect — malicious instructions hidden in content the model ingests: a web page, a document, a retrieved record, a tool’s output. The payload arrives through data the user never sees, which makes it far more dangerous and far harder to filter. An agent that reads a booby-trapped web page can be hijacked by that page.

Defenses reduce risk without eliminating it, and clarity about that gap is itself a security posture:

• Filtering catches known patterns and is bypassable — a model-based filter is itself injectable, so it can’t be the only layer.
• Privilege separation is the durable answer, and it moves the problem out of the prompt entirely: assume injection will succeed and constrain what a hijacked model can do. This is the pivot from “prevent bad input” to “limit blast radius,” and it’s why the real control is authorization, below.
• Human confirmation on consequential actions defangs injection that reaches a dangerous operation — the SOAR gate again.

The mental shift that matters: stop trying to guarantee the model won’t be tricked, and design so that being tricked isn’t catastrophic. Injection prevention is best-effort; blast-radius limitation is where the security actually lives.

Tool-use authorization is the real control

The most important security decision in an agentic system is not what the model might be made to say — it’s what it is authorized to do. An LLM with tools is an execution surface, and its permissions are the security boundary.

This reframes the whole problem as access control:

• Least privilege for agents. An agent should hold the minimum tools and permissions for its task, and no more. A research agent that only reads cannot be injected into deleting, because it has no delete tool — the capability simply isn’t in reach.
• Scoped, short-lived credentials. An agent’s authority should be bounded and expiring, the same temporary-over-standing logic as any workload identity.
• Separation of duties across agents. In multi-agent systems, the agent exposed to untrusted content should not be the one holding write access — a boundary that contains a compromise rather than letting it propagate.
• The confused-deputy framing. An agent acting with its authority on attacker-influenced input is exactly the confused deputy, and the classic mitigation — don’t grant the deputy more authority than the request should carry — applies directly.

This is the point the IAM frameworks page and this one converge on from opposite directions: agent security is an authorization problem, and “what is it allowed to do, scoped how tightly, revocable how fast” is the question, not “how do we phrase the system prompt.”

OWASP Top 10 for LLM Applications

OWASP maintains a Top 10 for LLM applications, worth using as a checklist with the caveat that it’s evolving quickly (see the caution above). The durable themes, independent of the current numbering:

• Prompt injection (direct and indirect) — covered above.
• Insecure output handling — trusting model output as safe. If a model’s output is rendered as HTML, run as code, or executed as a query, it’s untrusted input to that sink and needs the same encoding and validation as any user data. The model is not a trusted source.
• Excessive agency — too much tool access, the authorization problem above.
• Sensitive information disclosure — training-data or context leakage, including across a tenant boundary.
• Supply chain — below.

Verify the current list rather than trusting this summary; the value is the checklist habit, not the specific entries.

Model supply chain

The supply-chain problem has a new surface. You increasingly run models, adapters, and datasets you didn’t produce, and the trust questions are unfamiliar:

• Model provenance. A model downloaded from a hub is code and weights from a third party. Where did it come from, and what’s your basis for trusting it? Serialized model formats can execute code on load — the same deserialization risk, new file type.
• Poisoned models and datasets. Training or fine-tuning data can carry backdoors and biases that survive into behaviour, and they’re hard to detect by inspection.
• Dependency risk. The ML stack is a deep, fast-moving dependency tree with the same exposure as any other — plus its own less-mature tooling.

Treat models and their data as supply-chain artifacts with provenance, integrity, and vetting requirements, not as trusted givens because they came from a well-known hub.

Where this connects

This is the defensive counterpart to LLM pentesting and the application-side view of agent identity — the two pillars meeting at authorization. Insecure output handling and model supply chain are AppSec problems in new clothing. Blast-radius limitation is the SOAR guardrail applied to agents. And data leakage is data-privacy — a model and its context are data stores with audiences.