Stop Pressing 1: Measuring Human Rubber-Stamping in Agent Oversight

Abstract

Human-in-the-loop approval is a common mitigation in agentic security and one of the least measured. Agent frameworks terminate consequential actions: tool calls, commits, sends, installs with a confirmation prompt and assume the human will verify. In practice, approval and verification diverge: operators can habituate to confirmation dialogs, similar to warning fatigue in traditional security UX, and repeated low-cost approvals reduce verification effort over time. The result is a security boundary guarded by reflex rather than verification — one that can fail to stop prompt-injected tool misuse, hidden side effects, or action substitution. This poster presents a threat model (approval ≠ verification), a measurement framework for oversight quality, and an open-source harness that enables reproducible agent misbehavior. Before approving, the operator types a plain-language prediction of the agent’s intended action. Oversight fidelity is measured as the difference between predicted and executed behavior and ground truth, not the agent’s self-description. We compare passive confirmation, independent prediction gates, and agent-generated verification prompts and test whether verification prompts themselves become attack surfaces when generated by compromised agents. Seeded misbehaviors include hidden parameter changes, action substitution, omitted side effects, misleading summaries, and dependency substitution attacks. We track how often operators catch seeded misbehaviors versus how often they flag clean requests.