On May 20, 2026, InfoQ published OpenAI Outlines WebRTC Architecture for Low-Latency Voice AI at Scale. OpenAI revealed that to deliver realtime voice AI globally, it replaced traditional media termination models with a "relay transceiver" design. This achieves an architecture where WebRTC session state is isolated within dedicated transceivers, integrating smoothly with Kubernetes and cloud load balancers.
For those of us building voice AI infrastructure for mid-sized enterprise clients, this means "a reference architecture has finally arrived that lets us rebuild client voice projects—previously sustained through WebSockets and custom signaling—around industry standards." While previous articles on OpenAI Voice API + Parloa Service Agent Development for Clients and Low-Latency Agent Development for Clients via OpenAI WebSocket Execution addressed the product and application layers, the definitive answer for the infrastructure layer has now been presented. This article outlines our "WebRTC Realtime Voice AI Infrastructure" client engagement package.
Why voice AI hits a ceiling with WebSockets
| Dimension | WebSocket (traditional) | WebRTC relay transceiver (OpenAI approach) |
|---|---|---|
| Latency | 200〜400ms | 50〜150ms |
| Media optimization | Application layer implementation | Standard ICE / DTLS / SRTP |
| Load balancer affinity | Sticky sessions required | Made stateless via relay design |
| Kubernetes compatibility | Disconnects on Pod restarts | State isolated in transceivers |
| NAT / firewall traversal | Port forwarding required | Automatic punching via ICE |
| Echo cancellation | Application implementation | Standard on browser side |
| Codecs | Arbitrary (heavy) | Opus / standard |
| Scaling ceiling | Constrained by load balancer | Scale-out at relay tier |
In short, because the WebRTC relay model "surpasses WebSockets across latency, scale, and operability," it is set to become the future standard for production-grade enterprise voice AI architectures.
Three structural shifts driven by OpenAI's WebRTC relay design
Structural shift 1: From doing everything in the application to delegating the media tier to WebRTC
In past client voice AI projects, having the application layer handle audio buffering, retransmission, and echo cancellation was common, which caused code bloat and made maintenance difficult. By delegating these to WebRTC, applications can focus exclusively on text-centric AI logic.
Structural shift 2: From sticky sessions to scale-out at the relay tier
With WebSockets, connections are pinned to specific Pods holding the session, causing friction with Kubernetes and frequent call dropouts during Pod replacements. OpenAI's relay transceiver decouples the tier that exclusively handles media relaying, allowing the AI inference layer to remain completely stateless.
Structural shift 3: From single-region deployments to global media relays
Because WebRTC enables TURN servers and relays to be geographically distributed, you can construct an architecture routing from the relay closest to the user to the AI inference layer. This aligns with the global edge philosophy discussed in Cloudflare Dynamic Workflows Multi-Tenant Client Development.
Five phases of the "WebRTC Realtime Voice AI Infrastructure" delivered for clients
Phase 1: Current state assessment (2 weeks)
- Inventory of existing voice AI, IVR, and call center systems
- Communication protocols currently in use (WebSocket, SIP, custom)
- Latency measurements (end-to-end, inference segment, network segment)
- Concurrent connection counts + behavior during peak loads
- Infrastructure configuration (Kubernetes, ECS, VMs)
Phase 2: Architecture design (2–3 weeks)
- Selecting the WebRTC signaling layer (custom, Daily, LiveKit)
- Geographic placement of relay tiers (TURN / SFU)
- Connection interface with the AI inference layer (gRPC, WebRTC DataChannel)
- Selection of LLM, TTS, and STT models
- Failover design (relay tier and inference tier independently)
- Phased migration plan from existing systems
Phase 3: PoC construction (3–5 weeks)
- Minimal configuration: one relay tier + one inference tier
- Latency benchmarking (target: < 200ms end-to-end)
- Concurrency testing (target: 100 concurrent streams)
- Quality evaluation (MOS, WER, disconnect rate)
- Fault injection testing (relay tier outage / inference tier outage)
Phase 4: Production rollout + global distribution (3–4 weeks)
- Phased rollout (dev → staging → production canary → full production)
- Deploying relays across multiple regions (Tokyo, Singapore, Osaka, etc.)
- SLO dashboards in Datadog and Grafana
- Creating incident response runbooks
- Operational handover to the client SRE team
Phase 5: Monthly Quality Reviews (Ongoing)
- Monthly reporting on latency, WER, and MOS
- Disconnect rate tracking and root cause analysis
- Model upgrade evaluations (OpenAI, Anthropic, Google)
- Relay tier cost optimization
- User inquiry aggregation and improvement proposals
Standard technology stack set for custom development
| Layer | Recommended technology | Alternative |
|---|---|---|
| WebRTC signaling | LiveKit / Daily | Custom + Pion |
| TURN / SFU | LiveKit Cloud / Cloudflare Calls | Self-hosted Coturn |
| STT | OpenAI / Gemini 2.5 Flash / Whisper | Deepgram |
| LLM | GPT-5.5 / Claude / Gemini 3.5 | Self-hosted Llama 4 |
| TTS | OpenAI / ElevenLabs / Gemini | VOICEVOX |
| Orchestration | Kubernetes + Istio | ECS Fargate |
| Observability | DataDog / Grafana / Sentry | New Relic |
| Alerting | PagerDuty / Slack | Opsgenie |
Which projects need this and which do not
| Projects requiring this | Projects not requiring this |
|---|---|
| Running call centers / IVRs in production | Text chat only |
| Voice AI with 100+ concurrent connections | Development and validation stage |
| Disconnections and latency directly impact CSAT | Non-realtime recording and playback |
| Global deployment across multiple regions | Single-location operation |
| Operates a Kubernetes environment | Legacy PBX only |
Six clauses to include in client contracts
| Clause | Details | What the client should verify |
|---|---|---|
| Latency SLA | Upper limits for end-to-end / inference segments | UX impact scope |
| Disconnect rate SLA | Monthly / regional breakdown | Business impact level |
| Scaling ceiling | Guaranteed concurrent connection threshold | Billing terms for peak overages |
| Quality metrics | MOS / WER measurement frequency | Agreement on evaluation methodology |
| Model selection responsibility | Responsibility for selecting LLM, STT, and TTS | Client approval process |
| Offboarding handover | Architecture diagrams + rules + dashboards | Internal operational continuity |
Client ROI estimate (assumes 60-seat call center / 100 concurrent connections)
| Item | Existing IVR + human agents | After WebRTC voice AI implementation | Difference |
|---|---|---|---|
| Average response time | 90 seconds | 5 seconds | -85 seconds |
| First-tier automation rate | 0% | 60% | +60% |
| Staffing costs (annual) | 180 million yen | 90 million yen | -90 million yen |
| Infrastructure costs (annual) | ¥12,000,000 | 18 million yen | +6 million yen |
| CSAT (5-point scale) | 3.4 | 4.1 | +0.7 |
| Annual benefit | — | — | Approx. 84 million yen saved + improved CSAT |
Even for an enterprise production tier (initial 18 million yen + 9.6 million yen annually), the investment can be recouped within six months.
Five common pitfalls
Pitfall 1: Modifying WebSockets instead of migrating
Teams often propose tweaking WebSockets "just to shave off latency," but this sacrifices standardized media layers (ICE / DTLS / SRTP) and leads to unmaintainable code. Migrating to WebRTC is the proper approach.
Pitfall 2: Running TURN servers as a single instance
If the TURN server goes down, all calls drop. At minimum, a multi-AZ setup with automatic failover is mandatory.
Pitfall 3: Sharing the inference tier across multiple tenants
Routing inference requests for multiple clients through the same inference pool means a traffic spike from one client causes latency for others. Per-client namespaces + quota enforcement are essential.
Pitfall 4: Overlooking quality degradation during model switches
Switching models for LLM, STT, or TTS impacts WER, MOS, and latency across the board. Implementing A/B testing + phased rollouts is vital, echoing the Trust Layer philosophy discussed in Validating Agentic Behavior Client Development.
Pitfall 5: Assuming voice AI stands completely on its own
Voice AI delivers real business value only after connecting to existing CRMs, ticketing systems, and knowledge bases. You should incorporate business system integration via MCP servers into your design from day one (see Migrating Existing APIs to MCP Servers).
90-day action plan
| Week | Action |
|---|---|
| Week 1〜2 | Inventory of existing voice systems + requirements definition |
| Week 3〜5 | WebRTC architecture design + signaling selection |
| Week 6〜8 | PoC construction + latency/quality benchmarking |
| Week 9〜10 | Production rollout + failover validation |
| Week 11〜12 | Business system integration (CRM / knowledge base) |
| Week 13 | Production cutover + establishing monthly review meetings |
Summary — Aligning voice AI infrastructure with industry-standard architecture
OpenAI's WebRTC relay transceiver design serves as the industry reference for realtime voice AI infrastructure. For partners supporting mid-sized enterprises, our "WebRTC Realtime Voice AI Infrastructure"—integrating inventory, architecture design, PoC, production deployment, and monthly reviews—represents the new standard service offering.
Whether your WebSocket-based voice AI has hit a ceiling, you want to deploy a call center AI into production, or you need to deliver consistent quality across global branches, feel free to reach out via our contact form.
Sources
- OpenAI Outlines WebRTC Architecture for Low-Latency Voice AI at Scale（InfoQ）
- OpenAI Voice API + Parloa Service Agent Development for Clients (GH Media)
- Low-Latency Agent Development for Clients via OpenAI WebSocket Execution (GH Media)
- Cloudflare Dynamic Workflows Multi-Tenant Client Development (GH Media)
- Migrating Existing APIs to MCP Servers (GH Media)
