Skip to content

Security

Patterns and techniques for building agents that resist manipulation, protect sensitive data, and fail safely.

Threat Models

Threat models identify the structural conditions that make agent systems exploitable and prescribe architectural mitigations.

  • Action-Audit Divergence: A Four-Mode Taxonomy for Runtime Hardening — Name the four ways an agent action can diverge from its audit record (gate-bypass, audit-forgery, silent host failure, wrong-target) to convert "is this runtime hardened?" into a coverage checklist against existing controls
  • Compositional Vulnerability Induction in Coding Agents — Decomposing a malicious end-state into three innocuous engineering tickets bypasses refusal and hardening defenses at 53–86% ASR across nine production coding agents; pentester-framed reviewers close most of the gap
  • Constraint Drift: Why Safety Must Be Maintained, Not Asserted — Safety constraints encoded in prompts weaken across six trajectory surfaces — memory, delegation, communication, tool use, audit, optimization; the four-property invariant (fresh, inherited, enforceable, auditable) keeps them operative when delegation depth, memory persistence, and tool surface compose
  • Context-Fractured Decomposition Attacks on Tool-Using Agents — Attacks split across tools, modules, and time slip past defenders that only inspect a single contiguous conversation; artifact provenance gaps let benign intermediate steps recompose into a jailbreak downstream, lifting attack success by up to 28.3 percentage points
  • Distributed Cross-PR Attacks in Persistent-State AI Control — An untrusted coding agent spreads a covert payload across a sequence of PRs, timing each piece for natural cover, so stateless per-PR review never sees the whole attack; a stateful link-tracker plus a monitor ensemble cuts gradual-attack evasion from 93% to 47%
  • Forged Reasoning Trace Attacks on Agent Memory (FARMA) — Forged reasoning traces poison an agent's stored decision history so it skips a safety step believing it already ran; evasive wording slips past keyword filters and self-referential amplification defeats consensus defenses, reaching 100% success on binary safety-gate agents
  • Computer-Systems Lens for Always-On Agent Security — Model an always-on agent as a computer system — gateway runtime as OS, Skills as applications, Plugins as loadable extensions — to port classical OS protections onto the four cross-component surfaces that model-response benchmarks miss
  • Four-Layer Taxonomy of Agent Security Risks — Group threats into context/instruction, tool/action, state/persistence, and ecosystem/automation layers to map controls and surface coverage gaps where attacks propagate across boundaries
  • Goal Reframing: The Primary Exploitation Trigger for LLM Agents — A 10,000-trial taxonomy finds goal reframing — not social engineering or incentives — is the one prompt condition that reliably triggers vulnerability exploitation across models
  • Improper Output Handling: Validate Agent Output Before Downstream Use — OWASP LLM05 — agent output executed, rendered, or interpreted downstream without per-sink validation is an injection surface; enumerate the sinks (commit, exec, SQL, render, install) and gate each one
  • Lethal Trifecta Threat Model — Risk emerges when an agent has private data access, untrusted input, and egress simultaneously; remove at least one leg from every execution path
  • Oracle Poisoning: Knowledge Graph Corruption Against Tool-Using Agents — Corrupting a knowledge graph an agent queries via tool-use produces 100% trust at moderate attacker sophistication across nine models; the attack is distinct from prompt injection because the data path, not the instruction path, carries the payload
  • OWASP LLM Top 10 (2025): Agent Security Crosswalk — Map each OWASP LLM Top 10 (2025) risk to coding-agent-specific manifestations and site pages — a navigation aid for readers arriving with the framework's shared vocabulary, not a recommended threat model
  • Pre-Trust Execution Surface in Coding Agent Harnesses — Project-local config (settings files, hooks, MCP manifests, env vars, localhost listeners) executes before the trust prompt fires; defer parsing and execution until after the trust boundary is established
  • RAG Architecture as a Poisoning Robustness Decision — Under controlled knowledge-base poisoning, attack success rates span 24.4% to 81.9% across four RAG architectures with comparable clean accuracy; architecture choice is part of the threat model
  • Trojan Hippo: Dormant Memory Payloads Triggered by Sensitive Topics — A single untrusted tool call plants a dormant payload in agent memory that activates sessions later when the user discusses sensitive topics, exfiltrating data via outbound tools; tested defenses cut attack success to 0–5% but at steep utility cost
  • Unbounded Consumption: Bounding Agent Resource Use Against DoS and Denial-of-Wallet — OWASP LLM10:2025 framed as a same-surface, two-owner threat (availability and finance); the five complementary bounds — per-call token, per-task iteration, fan-out concurrency, cost-velocity, per-day dollar — close the cost dimension no single layer covers, with $46K/day Sysdig and $82K/48hr Gemini incidents as the empirical floor

Prompt Injection

Prompt injection is the primary attack vector for agents that consume untrusted content. External instructions embedded in web pages, emails, documents, or API responses can redirect an agent's behavior at the model level.

Anti-pattern: Single-Layer Prompt Injection Defence — Relying on one safeguard leaves agents vulnerable to attack vectors that layer does not address

Sandboxing

Isolation limits what a compromised or misbehaving agent can affect.

Anti-pattern: Hostname-Allowlist Proxy: The TLS-Inspection Blind Spot — A hostname-allowlist proxy without TLS termination enforces the client-supplied destination, not the actual destination; broad shared-CDN entries open domain-fronting and similar exfil paths

Data Protection

Preventing sensitive data from entering agent context is cheaper than scrubbing it after the fact.

Permissions

Excess permissions expand the blast radius of any failure or attack.

Code Injection

Code injection in multi-agent pipelines exploits agent trust in code it reads as input, distinct from prompt injection against a single agent.

Multi-Agent Propagation

Multi-agent systems with shared retrieval propagate adversarial content agent-to-agent. Defenses target the contagion channel and the per-agent detection signal.

PR-Time and Scheduled Review

Operational patterns that apply security agents to incoming changes and to resident codebase risk on different cadences.

  • Always-On Agentic PR Security Review — Pair a PR-time security reviewer with a scheduled whole-codebase scanner so new and resident risk both have continuous coverage; treat the reviewer agent itself as an injection target
  • Scanner-as-MCP-Server: Secret and Dependency Scans as Typed Agent Tools — Ship the security scanner as an MCP server so the agent invokes typed scans pre-commit and reasons over structured findings; qualified by five failure modes including agent-skips-scan and lethal-trifecta closure on the scanner principal
  • Cross-Repository Security Posture for Agent-Introduced Vulnerabilities — Treat one agent-introduced finding as a class to enumerate and remediate across every repository via variant analysis, not a per-repo point-in-time event; the organization-wide layer above per-agent scoping and per-PR review
  • Structural Monitoring for Covert Safeguard-Weakening — Diff the control- and data-flow graph of an agent's infrastructure-as-code change to catch a safeguard weakened under task success; an untrained structural monitor missed 3.5% of sabotage versus 11.6% for a git-diff baseline and cut joint sabotage success to 0% under synchronous rollback

Tool Invocation

Tool invocation exposes attack surfaces distinct from prompt injection. Malicious tools exploit argument generation and return processing to leak context and execute arbitrary commands.

Supply Chain

Agents dynamically load tools from MCP servers, plugins, and registries at runtime. A tampered tool inherits the agent's full permissions.

  • Agent-Emitted Dependency Version Ranges Widen the Supply-Chain Attack Surface — Agents default to caret and tilde ranges because npm install does; for an application with a bump-bot, replace the range with an exact pin plus a lockfile-enforced install — the floating range is the leg that admits a future-compromised release
  • Supply-Chain Security Debt in Agent Pull Requests — Across 4,022 agent-authored PRs, 82.3% of security smells are supply-chain integrity issues (unpinned CI/Docker references) concentrated in workflow and Docker files; target review at high-impact infra paths and enforce SHA pinning plus secret scanning in CI
  • Content-Addressed Agent Configurations (Deterministic Control Plane) — Treat coding-agent configs as an installed supply chain with SHA-256 content addressing, a per-project lockfile, and five declared permission tiers; the 10.1% cross-repo duplicate rate and the <1% configs declaring any permission scope (vs 33% of GitHub Actions workflows) make this a governance gap, not a hypothetical one
  • LLM-Pinned Library Versions Carry Systemic CVE Exposure — Across 10 models on 1,000 Python tasks, 36.7%-55.7% of LLM-specified versions contain known CVEs and all models converge on the same risky releases — pin against an external vulnerability source, not the model's training prior
  • Skill Composition Risk in Agent Ecosystems — Skills benign in isolation become harmful when one skill's output flows into the next; three failure modes (CapFlow, TrustLift, AuthBlur) reach 33.6%, 96.5%+, and 71.8% relative attack success across ten production backends, and per-skill vetting misses them by construction
  • Skill Supply-Chain Poisoning — Malicious skills injected into public registries exploit in-context learning to execute payloads hidden in documentation examples, bypassing alignment that blocks explicit instruction injection
  • Setup Documentation as an Install-Time Attack Vector — Setup docs are unverified install authority; editing only a README, requirements file, or Makefile redirects an agent's install to a wrong name, untrusted registry, or vulnerable version, and the payload runs at install time — agents catch typosquats but miss source-based redirects almost everywhere
  • Slopsquatting: Hallucinated Package Names as a Supply-Chain Vector — Coding LLMs hallucinate package names at 5.2%-21.7%; 43% of those names persist across re-runs, making them enumerable — attackers register the persistent names and the agent's install step pulls the malicious package
  • Tool Signing and Signature Verification — Require cryptographic signature verification (Sigstore/Cosign) before an agent loads or invokes a tool

Defense in Depth

No single safety mechanism is sufficient. Layered defenses ensure that failure of one layer does not compromise the agent.

Economics

Sizing frames for pre-release security review when vulnerability discovery scales with inference spend.

  • Security Budget as Token Economics — Treat hardening as a budget-allocation decision: AISI's Mythos evaluation shows no diminishing returns inside 100M tokens per attempt, but the outspend frame applies only where the search curve is still climbing and triage capacity absorbs findings

Deployment Models

Release patterns for capabilities whose offense-defense asymmetry makes broad release the wrong default.

Last reviewed: 2026-05-27