Stop Pressing 1: Do Students Verify Before They Approve? A Pilot Study of Comprehension Gates in Human Oversight of AI Agents

Abstract

Human-in-the-loop approval is a common mitigation in agentic security and one of the least measured. Agent frameworks terminate consequential actions: tool calls, commits, sends, installs with a confirmation prompt and assume the human will verify. In practice, approval and verification diverge: operators can habituate to confirmation dialogs, similar to warning fatigue in traditional security UX, and repeated low-cost approvals reduce verification effort over time. The result is a security boundary guarded by reflex rather than verification. Oversight fidelity is measured as the difference between predicted and executed behavior and ground truth, not the agent’s self-description. In the study, we compare passive confirmation, independent prediction gates, and agent-generated verification prompts and test whether verification prompts themselves become attack surfaces when generated by compromised agents. Seeded misbehaviors include hidden parameter changes, action substitution, omitted side effects, misleading summaries, and dependency substitution attacks. We track how often operators catch seeded misbehaviors versus how often they flag clean requests. A live, scripted demo is deployed now. Our pilot uses high school computer science classrooms in a controlled environment for studying oversight behavior under repeated approval tasks. The poster demonstrates where confirmation-based oversight succeeds, where it breaks, and how to measure the difference.