Beyond CTFs: Engineering AI Agents for Real-World Web Pentesting

Abstract

Large language models can solve CTF-style challenges, but production web pentesting is a different systems problem. Authenticated sessions expire, CSRF tokens rotate, single-page applications mutate the DOM, and request surfaces expand into thousands of near-duplicate states. Agents often exhaust their context before reaching meaningful attack paths.

This work presents the architecture of an autonomous browser-based agent for real-world web application pentesting. The system combines browser automation, DOM and accessibility-tree extraction, network capture, request normalization, page-similarity clustering, category-based memory, and task-specific reasoning loops.

Our central finding is that model selection is not the primary bottleneck; context engineering is. By controlling what the agent sees, remembers, and ignores, the harness can remove approximately 90% of redundant page states before model reasoning, reduce loops, and preserve useful evidence across authenticated workflows. The poster also proposes practical evaluation criteria: meaningful coverage, bug density, context cost, and loop rate. Pentest Copilot, the open-source implementation with 1.1K GitHub stars, demonstrates how these ideas move autonomous pentesting beyond CTFs and toward large, stateful production applications.