Policy driven agentic red teaming

Abstract

Tool-using AI agents are vulnerable to indirect prompt injection: adversarial instructions hidden in tool outputs, RAG retrievals, and poisoned memory that can steer an agent to exfiltrate PII, take unauthorized actions, or corrupt records. Red-teaming for these failures is largely manual where teams build synthetic environments and hand-place payloads. This scales poorly across real deployments, and generic benchmarks miss deployment-specific policy.

We present an end-to-end pipeline that turns the policy documents governing a deployment into artifacts a red-teaming framework can run. A policy mapper grounds each requirement to an evidence passage and maps it across four risk taxonomies; a scenario generator profiles the target system and produces concrete attacks as attack trees, actor profiles, and Gherkin specs; an artifact generator synthesizes a lean environment per scenario (deterministic codegen for schemas and tools, an LLM only for seed data) validated before any agent interaction. Red teaming frameworks that are unable to access the attack surface or observe the attack success behavior are flagged as unsupported for that scenario. Evaluation is fully deterministic, using predicate-based security checks and tool-call-sequence utility comparison. On a customer-service agent, the pipeline extracted 27 risks from one policy document and generated 31 scenarios across eight agentic threat classes, rendered for both Garak and AgentDojo. It extends to attacks such as denial of service, memory poisoning, and goal hijacking.