Updated September 28, 2026: We merged in the content of a related article, "Can you switch the AI you connected last year to a different model today?" (published September 24). We also added the date Agent Router joined AAIF; corrected its status before the move, the setup for running it yourself, whether a hosted version exists and agentgateway's affiliation in line with official sources; and removed unsupported frequency claims and rules of thumb for the number of API keys.
As the number of systems that use AI inside a company grows, separate API keys and separate invoices pile up for each application. For example, the sales support tool uses OpenAI, the internal help desk bot uses Claude, and a feature under evaluation uses Gemini. At that point, you can no longer say, from a single place, how much you are spending in total or which data is being sent to which model.
The awkward part is that each application's code has its provider's SDK and endpoint written directly into it. Even when someone says, "Costs are high, so let's switch to another model", the work starts with finding the places that make the calls.
The tool that brings order to this state is what is called an AI gateway. In September 2026, Agent Router (formerly Envoy AI Gateway), an open-source project in this space, became a project of the Agentic AI Foundation (AAIF) under the Linux Foundation. Using Agent Router as an example, this article sorts out what adding one layer in between solves and what it leaves, whether to run it yourself or leave it to a provider, and what to decide before introducing one.
Note that this article covers traffic in which internal applications and agents call models through APIs. Chat services that employees use in a browser do not normally go through this path, so managing their subscriptions and usage needs to be considered separately.
What exactly does an AI gateway do?
What it does is simple: place one layer between your applications and the AI models, and route all traffic through it. The benefits follow from having done that. Following Agent Router's official description, they are as follows.
- There is one place to connect to. Applications no longer hold an SDK or endpoint for each model; they talk only to the gateway. According to Agent Router's description, you connect to every model through a single OpenAI-compatible API, and once the providers behind it are configured, switching models is just a matter of changing the model name in the request
- You can see who used how much. Because everything passes through one place, you can get usage by team, application and model, and set limits as well. In Agent Router, limits are set in units of tokens, such as input, output or total tokens
- Keys move out of your applications. Applications authenticate only to the gateway, and each provider's API keys and cloud credentials are managed and rotated together on the gateway side
- You can record what was sent. Which model answered, how many tokens were used and which tools were called can all be recorded without changing your applications. However, stripping sensitive information from prompts before they are sent is not a standard Agent Router feature; it is positioned as something you add through Envoy's extension mechanisms
Supported destinations include OpenAI, Anthropic, Amazon Bedrock, Azure OpenAI, Google's Gemini and Vertex AI, Mistral, Cohere, DeepSeek, Together AI and Groq. You can also connect self-hosted models, as long as the server responds in the OpenAI format, as vLLM does.
Agent Router does not only handle connections to models. Connections to MCP tools go through the same layer. In the official site's words, from an agent's point of view it is "one OpenAI-compatible API for every model and one router for every MCP tool". It combines the tools of multiple MCP servers into a single catalog and narrows down the tools it shows depending on who is calling. If you publish internal systems as MCP servers, being able to decide in one place which tools each agent can see starts to pay off. MCP itself is covered in the complete guide to MCP.
Envoy AI Gateway became Agent Router
In an announcement dated September 9, 2026, AAIF said that Envoy AI Gateway was joining it under the new name Agent Router. According to the project's own announcement, the official join date is September 10.
Agent Router is an open-source AI gateway built on Envoy Proxy and Envoy Gateway. It was started by engineers at Bloomberg and Tetrate and has been developed in the open since October 2024. With version 1.0 in June 2026, its configuration API became stable, and the project commits to compatibility across the 1.x series (1.1 followed in August).
Even before the move, it was a subproject of Envoy, which is developed under the CNCF, and not a single company's product. What changed is that it went from being an Envoy subproject to a standalone AAIF project. The name, repository and website have moved, but the code, maintainers, license (Apache 2.0), configuration format and commands are unchanged, and existing deployments are said to need no migration work.
AAIF is a neutral foundation whose formation the Linux Foundation announced in December 2025, with MCP, goose and AGENTS.md as its first projects. It now also hosts the Agent2Agent (A2A) protocol, among others.
According to AAIF's announcement, 11 organizations have publicly disclosed their adoption, including Bloomberg, Tetrate, Tencent Cloud, Nutanix, LINE Yahoo (LY Corporation) and the National Research Platform. It says the nine maintainer seats are spread across Bloomberg, Nutanix, AMD, Tetrate and Netflix, with no company holding a majority. When considering whether to build it into business systems you will use for a long time, this kind of development structure is one factor in the decision.
There is a separate project with a similar name: agentgateway. It was created by Solo.io, accepted by the Linux Foundation in August 2025, and is now also listed among AAIF's projects. Since you cannot tell the two apart by the foundation they belong to, it is safer to specify both the project name and the URL when discussing technology selection internally.
What adding a layer in between solves, and what it leaves
Adding a gateway does not mean you can switch freely. Some things become a configuration change, and some remain even with a layer in between.
| What becomes a configuration change with a layer in between | What remains even with a layer in between |
|---|---|
| Switching the model or provider you call | Adjusting how prompts are written to suit the model |
| Absorbing differences in API formats and authentication between providers | Fixes on the receiving side when the output format or quirks change |
| Failing over to another provider during an outage | Redoing the evaluation to judge whether it is usable for the business |
The right-hand column is what really matters. How prompts work and how tool calls behave differ from model to model, so there is no guarantee that the same instructions will produce the same quality. What a gateway reduces is "the work needed to switch", not "the decision on whether switching is acceptable". How to read cost and quality when you use multiple models for different tasks is covered in the cost and quality trade-off of running multiple models.
Run it yourself, or leave it to a provider?
Do you run an open-source gateway yourself, or use a service operated by a provider, such as a cloud provider's AI gateway? This is the first fork. Agent Router itself has a hosted version run by Tetrate (Tetrate Agent Router Service), which is presented on the official site. Commercial versions with support that you place in your own cloud or on premises (products from Tetrate and Nutanix) are also featured.
| Run it yourself (open-source Agent Router and similar) | A provider's service (hosted versions and similar) | |
|---|---|---|
| Getting started | To try it, you can start it on a laptop with one command. The official production setup combines it with Envoy Gateway on Kubernetes | Sign up, receive an API key and switch your applications' endpoint |
| Suits you when | You already run Kubernetes. You want to keep logs and keys under your own control | You have few dedicated infrastructure staff. You want to make usage visible first |
| Effort | Building, updating, monitoring and redundancy all take effort | You can start from configuration |
| Important precautions | If it goes down, all the AI features connected to it stop at once | Prompts and responses also pass through the provider's infrastructure. You are exposed to the provider being acquired, discontinuing the service or changing its specification |
The deciding point is whose infrastructure your prompts and responses pass through, and where the logs are kept. With a provider's service, the content passes through the gateway provider in addition to the model provider, and the records are kept there too. If you have a requirement to avoid this, or to keep logs inside your own company, the effort of running it yourself is what you budget as the price. However, if the model itself is an external service, the prompts reach the model provider whichever option you choose.
For a company without dedicated infrastructure staff, we think it is realistic to start by making usage visible with a provider's service. An example of starting from usage limits is covered in setting usage limits in Cloudflare AI Gateway, and what changes when the provider you use as your gateway is acquired is covered in recounting dependencies in LLM procurement.
Decide these before you introduce one
Once you introduce a gateway, your company's AI use converges on that single point. Before selecting the technology, settle the following three.
- What happens when it goes down. Agent Router can be configured to route requests to another provider when a model provider goes down. But if the gateway itself stops, everything behind it stops. Decide in advance, for each business process, whether each application has a fallback path that calls the model directly or whether it is acceptable for it to stop
- How much logging to retain. When you enable tracing in Agent Router, the default format (OpenInference) records the full text of requests and responses. You can also exclude inputs and outputs from the records through configuration. Retaining the full text makes the log itself a concentration of sensitive information. Decide the scope, storage location and retention period before you start recording
- Who sets the limits. Are per-department limits set by the IT team, or requested by each department? Unless you also decide who to contact when a limit is hit and the procedure for raising it, coordination will be needed every time something stops
If your motivation for introducing one is avoiding vendor lock-in, also check whether the gateway itself becomes a new form of lock-in. Agent Router is open source under Apache 2.0, and the reported absence of any company holding a majority of the maintainer seats is reassuring. On the other hand, while calls from the application side are OpenAI-compatible and easy to move, the gateway's configuration is written in a format specific to Agent Router (resources such as AIGatewayRoute), so moving to a different gateway requires rewriting it. The procurement and contract side of this is covered in how to procure software without vendor lock-in.
Is it time to introduce one? Three questions to measure where you stand
If only one feature in your company calls AI, the benefit of introducing a gateway is small, and writing a thin layer that consolidates the calls yourself is enough. The judgment changes when more features call AI and usage starts to differ by department. Before deciding whether to introduce one, measure where you stand with the following three questions.
- How many places in the entire codebase specify model names and endpoints? How many API keys have been issued, and where is each one used?
- Can you say how much each feature spent last month without cross-checking the invoices?
- Do you have a procedure at hand for comparing your current model side by side with another model using the same inputs?
If you get stuck on all three, gathering the calls into one place will do more good than introducing a gateway. Once you are in that state, you can choose again later whether to move to Agent Router, a provider's service or your own layer.
On September 28, 2026, we checked AAIF's announcement, technical blog post and project page; Agent Router's official site, announcement and documentation (supported providers, setup steps, token limits and tracing); the README and license on GitHub; and the Linux Foundation's announcements of AAIF's formation and of agentgateway. We have not deployed Agent Router or the hosted version, tested how they work or measured their performance. We have also not checked the hosted version's pricing or terms of service, or how its features differ from cloud providers' AI gateways. Supported providers and features change over time, so check the official documentation when deciding whether to adopt it.
For consultations on organizing your internal AI connections, policies for managing usage and logs, or migrating existing applications, please get in touch through the contact form.
Sources
- Envoy AI Gateway becomes Agent Router and joins AAIF — Agentic AI Foundation
- Agent Router: Powerful traffic handling for agent builders — Agentic AI Foundation
- Agent Router — Agentic AI Foundation
- About — Agentic AI Foundation
- Envoy AI Gateway is becoming Agent Router, an Agentic AI Foundation project — Agent Router
- Agent Router official site
- Supported AI Providers — Agent Router
- Getting Started with Agent Router — Agent Router
- Usage-based Rate Limiting — Agent Router
- GenAI Distributed Tracing — Agent Router
- theagentrouter/agent-router — GitHub
- Linux Foundation Announces the Formation of the Agentic AI Foundation (AAIF) — The Linux Foundation
- Linux Foundation Welcomes Agentgateway Project to Accelerate AI Agent Adoption — The Linux Foundation
