In May 2026, OpenAI published new models for voice intelligence in Advancing voice intelligence with new models in the API. Around the same time, Parloa builds service agents customers want to talk to was released, highlighting how German voice AI startup Parloa is automating customer support for major enterprises with service agents that "customers want to talk to."
Until now, custom voice AI projects centered on TTS (text-to-speech) and STT (speech-to-text), but the new API has reached the level where real-time, bidirectional conversational agents can be built with low latency. This article outlines the new model for call center development projects for clients.
What has changed — Four advancements overcoming the "TTS wall"
| Item | Legacy generation (separate TTS / STT) | New generation of May 2026 |
|---|---|---|
| Latency | 800ms to 2s (round trip) | 200 to 400ms (speech-to-speech) |
| Interruption handling | Not supported (IVR-like) | Natural interruption and resuming after acknowledgments |
| Emotion and tone | Monotone readout | Tone modulation for empathy, disappointment, and delight |
| Multi-turn memory | Prone to losing context | Natural retention across dozens of turns |
In particular, "interruption handling" eliminates the fatal flaw of traditional IVRs in call center operations where "users cannot proceed until the system finishes reading the entire prompt," serving as the cornerstone for creating the "feeling of speaking with an actual human."
This extends beyond the "multilingual TTS" covered in Building 70-language voice customer experiences with Gemini Flash TTS, and is best understood along the evolution vector of moving from TTS to conversational agents.
Lessons from Parloa — Five principles of service agent design
The design principles published by Parloa are essential conditions for making agents that "customers want to talk to" when building voice agents in custom development.
Principle 1: Build "human escalation capability" first
Rather than aiming for full automation, incorporate a path from the start so that "users can connect to a human immediately when stuck." This is a prerequisite for customer satisfaction, and many operational designs set a "target escalation rate of 30%."
Principle 2: Incorporate "tolerance for silence"
Human operators tolerate silence, but AI tends to interpret silence as "completion." Training it that "3 seconds of silence = thinking" so that it does not rush the conversation is key to building trust.
Principle 3: Do not aim for one-call resolution
If you obsess over "resolving within a single call" for complex inquiries, the AI will make forced judgments and fail. Delivering value by simply "gathering information and passing it accurately to a human" is sufficient.
Principle 4: Log customer emotions
Structured logging of emotional changes during calls (such as frustration to satisfaction) creates a review pipeline for "calls ending in dissatisfaction." Much like the site analysis in GA4 best practices, visualization serves as the starting point for improvement.
Principle 5: Design with model switching in mind
Because voice models from OpenAI, Google, and Anthropic evolve every six months, insert an abstraction layer with a "ban on hardcoding specific models." This follows the same philosophy of "provider abstraction" discussed in the Private MCP server implementation guide.
Reference architecture to build in custom development projects
[顧客電話] ──→ [PSTN / SIP Gateway]
│
▼
[音声フロー制御層] ← Parloa 相当のオーケストレーション
├ 割り込み制御
├ ターン管理
└ エスカレ判定
│
▼
[音声-音声モデル層] ← OpenAI Realtime API / Gemini Live
│
▼
[業務 API 層] ← 注文照会 / 予約変更 / 在庫確認
│
▼
[監視・ログ層] ← 感情ログ / 通話録音 / KPI 集計By isolating the voice flow control layer, you can protect business logic even when swapping out voice models. This is "the essence of Parloa," and serves as a useful reference when building in-house.
Voice agent KPIs to include in client contracts
| KPI | Target value (typical) | Measurement method |
|---|---|---|
| Escalation rate | 20% to 30% (depending on workflow) | Automatically tallied from call logs |
| Average handle time | Equal to or less than existing IVR | Automatically tallied from call logs |
| Negative emotion rate | 5% or less | Sentiment evaluation by LLM |
| Repeat contact rate | 15% or less | Follow-up contacts from the same customer within 7 days |
| Accuracy rate | 90% or higher | Sample audits + customer feedback |
By including "accuracy rate" in contract KPIs, you can measure outcomes that are meaningful to business operations. Relying solely on speech recognition rate as a KPI overlooks failures where "the transcript is accurate, but the customer is angry."
Four common pitfalls
Pitfall 1: Over-investing in "natural conversation"
Aiming to make it "indistinguishable from a human" causes development costs to explode without delivering ROI. Aiming for "clearly better than IVR" is sufficient.
Pitfall 2: Call recording retention requirements
Call recordings are subject to personal data protection laws and industry regulations (financial and medical). Failure to explicitly specify retention periods, encryption, and access privileges in contracts risks fatal fallout from a "recording leak."
Pitfall 3: Quality degradation during model changes
Voice models from OpenAI and Google are updated every six months, and in rare cases, quality may regress. Incorporating an A/B testing mechanism from the start is key to maintaining operations.
Pitfall 4: The illusion of "zero waiting time"
Cloud voice API latency is the sum of network and model inference, meaning a 200ms latency cannot be guaranteed 100% of the time. Write realistic metrics into the SLA, such as "average 400ms, 95th percentile 1s."
Summary — Moving from the "TTS era" to the "conversational agent era"
With the new models in OpenAI's Voice API and Parloa's case studies, "real-time bidirectional voice service agents" have reached the stage of viable production deployment in custom development. While multilingual TTS remains valuable, the foundation for creating experiences "customers want to talk to" has shifted to conversational agent design.
A voice agent's overall architecture varies entirely based on the number of assigned scenarios, integrated business APIs, human staffing for escalations, and regulatory requirements governing call recordings. Discussions rarely proceed by fitting into a rigid template. As stated in Principle 1 of this article, the first thing to establish is where to draw the line between "what inquiries to entrust to AI and where to hand over to humans." If you are considering replacing an IVR or automating a multilingual call center, we can help you work through that boundary definition. Please share your current call flows using our inquiry form.
