AI Phone Receptionist Hardening
Structural improvements to an AI phone receptionist system for an insurance agency — fixing edge cases that only appeared with real callers and building test infrastructure so regressions are caught before production.
The challenge
An insurance agency had an AI phone receptionist built on Twilio telephony that could handle inbound calls — answer questions, qualify leads, gather quote info, and transfer complex cases to human agents. The system was already live, but real-world callers exposed a suite of problems the developers hadn't anticipated.
The AI skipped intake steps and transferred prematurely. It re-asked fields the caller already answered. Compound answers like "4312 Maple Lane, Dallas, Texas 75219" were treated as one field instead of four extractable values. Barge-in detection was slow (~800-1500ms), making callers wait awkwardly. System prompts ballooned to 8k-13k characters per turn. And there was no way to test fixes deterministically — every change meant manually calling the number and hoping for the best.
What was improved
I joined mid-project to harden the call flow, fix real-world edge cases, and build systematic testing infrastructure.
Phase-gated tool availability
The system was restructured so the AI's available actions change based on call phase. During greeting, it can only confirm intent and save caller details. During quote gathering, it can only record quote fields. Later-stage tools are unavailable until the required phase conditions are met, preventing the model from invoking those actions prematurely.
Barge-in detection
Response time when a caller interrupts improved from ~1 second to near-instant, making conversations feel natural instead of robotic.
Prompt efficiency
System prompts per turn were reduced by 60-70% through structured prioritization: essential context is always present, optional details are added only when relevant, and a validator warns when instructions conflict or exceed budget. Prompt length per turn fell significantly, reducing input-token usage and helping lower latency and per-call cost.
Sequential state management
Instead of overwhelming the AI with all fields at once, the system presents one field per turn. When callers give compound answers like "4312 Maple Lane, Dallas, Texas 75219," the system now records all four values instead of just the first.
Transfer safety gates
Before transferring to a human agent, the system ensures a minimum handoff packet is collected (caller name, phone, insurance type, coverage needs). A partial transfer mechanism allows the call through when most data is present. This prevents callers from reaching a human with zero context.
Phone test harness
A local testing infrastructure so common regressions can be checked before changes reach live calls:
- Smoke mode — Fast simulation for quick validation of individual changes
- Prod-like mode — Realistic scenarios with deterministic noise profiles (transcript fragmentation, frustrated callers)
- Replay mode — Ingest real call logs and run automated checks without live calls
Evaluators automatically detect common failure patterns: field repetition loops, blocked transfers, and data quality issues. Results are stored for comparison across runs.
The result
- Structural constraints, not persuasion -- later-stage tools are unavailable until phase conditions are met
- Application-side barge-in handling measured under 50 ms in testing; callers can interrupt naturally without awkward waits
- Compact prompts -- prompt length per turn fell by 60-70%, reducing input-token usage and helping lower latency and per-call cost
- Common call-flow regressions can be detected in the test harness before live deployment
This was a cloud-connected production system using Twilio for telephony and xAI Grok for model inference. My engagement covered call-flow reliability and testing, not local deployment or compliance certification.