For Prompt Injection, Press 1: Hacking AI Voice Agents

Abstract

What happens when you social engineer an AI agent that was trained to be helpful over the phone? Can you get it to reveal its system prompt out loud? Will it disclose information about other callers? How far can you push it before its guardrails kick in?

AI voice agents sit behind telephony layers like speech-to-text, text-to-speech, and call routing that introduce new attack surfaces and opportunities. They're replacing human operators everywhere: answering phones at doctor's offices, handling IT help desks, triaging customer support, booking appointments. They sound human, but underneath they're the same LLMs we've been prompt injecting.

I built an open-source tool that tackles this by placing real phone calls to voice AI agents, speaking attack scenarios using text-to-speech, capturing responses via speech recognition, and analyzing transcripts for signs of successful exploitation.