Ask “how much did we spend on AI last month?” and at some companies several different answers come back. What sales contracted for, what engineering uses for evaluation, what the IT team is trying out. Each on a separate API key, on a separate invoice, in a state where nobody knows how much is being spent in total.
If it were only about the money, accounting could add it up. The awkward part is that which data is being sent to which model has become equally unclear. And each department’s code has that vendor’s SDK written directly into it.
The tool that brings order to this state is what is called an AI gateway. In September 2026, one open-source project in this space moved into an industry standards framework.
What exactly does an AI gateway do?
What it does is simple: place one layer between your applications and the AI models, and route all traffic through it. The benefits follow from having done that.
- There is one place to connect to. Your applications no longer write a different SDK or endpoint per model. With Agent Router, you talk to every model through a single OpenAI-compatible API
- You can see who spent what. Because everything passes through one place, you can get usage by department and by purpose. You can also set limits
- You can see what is being sent. Which data leaves for external models can be recorded and inspected in one place
- You can swap models. You can change the model behind the scenes without touching application code
The fourth is the vendor lock-in argument. A caveat is needed, though: it is not as simple as “put a gateway in and you can swap models freely”. Prompt style and tool-calling behaviour differ by model, so swap the model and the quality changes too. What you remove is the connection overhead; it does not remove the need to re-verify quality. The practical reality here is organized in the cost and quality trade-off of running multiple models.
Envoy AI Gateway became Agent Router
In September 2026, Envoy AI Gateway joined the Agentic AI Foundation (AAIF) and changed its name to Agent Router. The code, the maintainers and the APIs carry over unchanged. Taken as just a rename it is not much of a story, but where it went matters.
AAIF is a body under the Linux Foundation that advances standardization of technologies around AI agents, including MCP, AGENTS.md and the Agent2Agent protocol. In other words, Agent Router has become a component held by a neutral body rather than a particular vendor’s product.
Agent Router does not only handle connections to models. Connections to MCP tools go through the same layer. From an agent’s point of view, it is one OpenAI-compatible API for every model and one router for every MCP tool. If you are heading towards publishing internal systems as MCP servers, this architecture should suit you well. MCP itself is covered in the complete guide to MCP.
Names cited as running it in production include Bloomberg, Tencent Cloud and Nutanix. At minimum, this is not a project stuck at the evaluation stage.
Note that there is a separate project with a similar name, agentgateway. That one was contributed to the Linux Foundation by Solo.io and is a different thing from Agent Router. It is easily confused in internal technology selection discussions, so it is safer to specify the URL as well.
Run it yourself, or use a managed service?
Do you run an open-source gateway yourself, or use a cloud provider’s managed service? This is the first fork.
| Run it yourself (Agent Router and similar) | Managed (offered by a cloud provider) | |
|---|---|---|
| Suits you when | You already run Kubernetes. You want logs and traffic paths to stay inside your own environment | You have one or two infrastructure staff. You want visibility into usage first |
| Effort | Build and operational effort is required. Knowledge of Envoy is needed | You can start with configuration alone |
| Important precautions | If it goes down, all internal AI use stops. Redundancy is a prerequisite | You are exposed to the provider discontinuing the service or changing its specification |
For a company of around 50 people with no dedicated infrastructure staff, managed is the more realistic starting point. The approach of starting from usage limits is covered in managing spending limits on an AI gateway.
Running it yourself is justified when you have the requirement that prompt and response content must not leave your own environment. If that requirement exists, the operational effort is what you should budget as the price of meeting it.
Decide these before you introduce one
The moment you introduce a gateway, it becomes a single point of failure for internal AI use. Before selecting the technology, settle the following three.
- What happens when it goes down. When the gateway fails, does each application have a fallback that calls the model directly, or is it acceptable for it to stop? Establish the scope of work that would halt, first
- How much logging to retain. Retaining full prompt text makes the log itself a concentration of sensitive information. Decide the scope and retention period before you start recording
- Who sets the limits. Are per-department usage limits set by the IT team, or requested by each department? Introduce it without deciding this and the IT team gets an inquiry every time someone hits a limit
If your motivation for introducing one is avoiding vendor lock-in, it is also worth checking whether the gateway itself becomes a new form of lock-in. Being open source held by a neutral body is one point of reassurance on that front. The framework for this decision is organized in how to avoid vendor lock-in.
What to do next
Start by counting the AI-related API keys issued inside your company. It is not unusual for the invoices on the accounting side and the number of keys actually issued to disagree. If the count is three or fewer, deciding who uses them does more than introducing a gateway. If it is over ten and you cannot explain who uses what for, you are at the stage of considering a layer in between.
GleamHub offers development, AI and automation consultations covering the architecture of internal AI platforms, policies for managing usage and logs, and migrating existing applications. Because the right architecture depends on your current usage and organization, please get in touch through the contact form.
Sources
- AAIF announces Agent Router joining — open source that brokers connections to AI models and MCP tools — gihyo.jp
- “Agent Router”, which absorbs and unifies API differences between AI vendors such as OpenAI and Anthropic, heads for industry standard status under the Linux Foundation — Publickey
- Agent Router — Agentic AI Foundation
- Agent Router official site
- Linux Foundation Welcomes Agentgateway Project — The Linux Foundation
