OpenAI WebSocket execution mode — Designing low-latency agents for clients in 2026

In May 2026, InfoQ published OpenAI Introduces Websocket-Based Execution Mode to Reduce Latency in Agentic Workflows, announcing that OpenAI launched a WebSocket-based execution mode for agentic workflows. By eliminating HTTP round-trips in favor of bidirectional streaming, tool call latency improved dramatically, making "agents requiring real-time responsiveness" a viable reality.

Among AI agent use cases, voice, chat, and call center integrations belong to domains where "waiting time directly determines perceived quality." In this article, we outline how to architect WebSocket Execution Mode in custom development, how to choose between it and HTTP mode, and key operational pitfalls.

What has changed — The structure of latency

The shift brought by WebSocket execution mode is simple yet critical: "waiting time differs even for identical processing."

In particular, "parallel tool execution" creates a massive difference in client projects where agents query multiple data sources concurrently. A flow that previously required waiting 1 second via sequential HTTP calls (5 tools × 200ms) can be compressed to 200ms via parallel streaming.

This is the latency counterpart to "evaluating adoption based on cost structures" covered in Engineering for custom development in the 4.5x cost era of Computer Use. It adds an option where "costs increase, but the user experience improves dramatically."

Criteria for choosing WebSocket vs. HTTP in custom development

This does not mean every project should use WebSockets. In custom development, evaluate selections using the following criteria.

[WebSocket を採用すべき案件]
  ├ 音声 / 動画リアルタイム連携
  ├ コールセンター / カスタマーサポート
  ├ ライブチャット (体感 1 秒以内必須)
  ├ 多数ツールの並列呼び出しが必要
  └ ストリーミング UI が前提

[HTTP で十分な案件]
  ├ バッチ処理 / 夜間ジョブ
  ├ 非同期メール返信生成
  ├ 報告書 / レポート自動生成
  ├ 単発のコード生成 (PR レビューなど)
  └ レイテンシ要件 > 2 秒 OK

In particular, "live chat" is actually a tricky area to evaluate; in many cases, HTTP is perfectly adequate if perceived latency can be masked with a "typing indicator." Measuring perceived latency through real-device testing is the standard workflow in custom development.

Standard custom development architecture — Four layers for WebSocket adoption

When implementing WebSocket execution mode in custom development, structure it into the following four layers.

[Layer 1: クライアント層]
  ├ ブラウザ / モバイルアプリ
  ├ 音声 SDK / WebRTC
  └ ストリーミング UI 描画

[Layer 2: WebSocket ゲートウェイ]
  ├ 認証 / 認可 / レートリミット
  ├ 接続管理 / heartbeat
  └ 障害時の自動再接続

[Layer 3: エージェント実行層]
  ├ OpenAI WebSocket Mode
  ├ ツール呼び出しの並列化
  └ メモリ / コンテキスト管理

[Layer 4: ツール / データソース層]
  ├ MCP サーバー
  ├ 業務 API / DB
  └ ベクトル検索 / RAG

In particular, connecting directly to OpenAI without hosting your own Layer 2 WebSocket gateway is not recommended in custom development from the perspective of authentication, auditing, and cost management. Placing a gateway in between is standard custom development practice.

This mirrors the "always insert an authentication layer" philosophy discussed in Implementing MCP server OAuth 2.1 authentication in custom development.

Latency measurement — Four KPIs agreed upon with clients

The latency KPIs defined in custom development contracts comprise the following four metrics.

In particular, "First Token Latency" directly impacts user experience as "dead air," and forcing a wait of even one second here almost certainly leads to drop-off. It is the most critical KPI to explicitly state in client contracts.

Five operational pitfalls

Here are common pitfalls to avoid when running WebSocket execution mode in custom development projects.

Pitfall 1: Forgetting to estimate connection maintenance costs

WebSockets maintain connections even during idle time. If you are planning for 10,000 concurrent connections, the scaling architecture of your gateway will be completely different from HTTP. Always perform load testing in advance.

Pitfall 2: Sloppy state recovery upon disconnection

When a WebSocket disconnects due to network interruptions, if you don't design for "how to restore the conversation continuity," users will end up having to retype their messages. Retain conversation state on the server side and synchronize it upon reconnection.

Pitfall 3: Concealing it with an HTTP-compatible wrapper

Attempting to "hide it behind HTTP because WebSockets are difficult" not only eliminates the advantages of WebSockets, but also leads to a hybrid hell of maintaining both modes. If you commit to using WebSockets, stick with WebSockets all the way through.

Pitfall 4: Insufficient observability design

While HTTP can be traced via request logs, WebSockets require a separate design for message-level logging. Link OpenTelemetry traces and spans to WebSocket messages.

This point aligns with the observability architecture discussed in Airbnb Observability Case Study: Migrating to OpenTelemetry; observability must be built into WebSockets from the start.

Pitfall 5: Managing long-lived connections on the backend

There are limits to holding tens of thousands of WebSockets in a single process on the server side. Select infrastructure specialized for connection management, such as Cloudflare Durable Objects or dedicated WebSocket gateway PaaS solutions.

Conclusion — Entering the era where "speed is a contractual deliverable"

With the advent of WebSocket Execution Mode, "agent speed" is no longer merely a tech stack choice, but a subject of contractual agreement. Specifying First Token Latency and disconnection rates as KPIs and evaluating performance based on actual measurements will become the new standard in custom development.

Whether you should switch to WebSockets ultimately cannot be decided without measuring what your current First Token Latency actually is in milliseconds. Furthermore, the subsequent steps—whether to self-host a gateway or rely on a PaaS, and how much of your existing HTTP implementation to retain—will dictate the required architecture and team structure. If you share your current measurements and existing configuration via our contact form, we can collaborate with you starting from assessing whether your architecture would benefit from adopting WebSockets.

Sources

• OpenAI Introduces Websocket-Based Execution Mode to Reduce Latency in Agentic Workflows（InfoQ）
• Engineering for custom development in the 4.5x cost era of Computer Use (GH Media)
• Implementing MCP Server OAuth 2.1 Authentication in Client Projects (GH Media)
• Airbnb Observability Case Study: Migrating to OpenTelemetry (GH Media)