AI gateway
architectureLLM gateway · model gateway · AI proxy
The single point all traffic between your applications and the models passes through: it authenticates the caller, applies policy, meters spend and leaves a trail. It's the proxy layer for AI.
Without a gateway, every team wires up its own model with its own key and nobody knows what it costs or what data left the building. With one, there's somewhere to pull the lever: swap providers without touching applications, cap a model, block a data type, set a spend ceiling per team, and answer the audit question of who called what and when. Security vendors already sell it under that exact name: CrowdStrike positions it as a centralised control point for enterprise AI traffic (SiliconANGLE, 2026-09-02). It's the cheapest piece of AI governance to stand up, and almost always the missing one.
How it differs from
- AI control plane
- The gateway is the checkpoint traffic crosses. The control plane also covers identity, policy and logging over what crosses it.
- Agent FinOps
- The gateway is where spend is measured and capped. FinOps is the discipline of deciding what's worth spending.
Examples
- A gateway that routes each request to the cheapest model meeting the quality bar and logs cost per team.
- A gateway filter that stops customer data from reaching an unapproved model.
FAQ
- What do I actually get from an AI gateway?
- Three concrete things: you can swap models without touching application code, you know what each team spends, and you have a call log that works as an audit trail. The first one pays for itself the day a model's price goes up.
- Can I build one without buying a product?
- Yes. A proxy of your own in front of the model APIs, with a key per team and request logging, covers most of the value. You buy a product when you need content policy and security controls you don't want to maintain yourself.