What an AI gateway does
Without a gateway, each application calls OpenAI, Anthropic or another provider directly, with its own key and its own way of logging. With a gateway, the applications call a single endpoint instead, and the gateway forwards each request to the right provider and model. On the way it can check who is calling, count tokens, stop a request that would break a budget, log what happened and pass the answer back. Many gateways offer an OpenAI-compatible API, so existing code and libraries keep working once the base address and the key are changed.
API keys in one place
Provider keys are the most sensitive part of an AI setup: whoever holds one can run up costs on your account. A gateway keeps the provider keys in one place, encrypted, and gives each application or team its own key to the gateway instead. If one of those keys leaks, it can be revoked without touching the provider key or the other applications. It also becomes clear which system made which calls, which matters when an invoice grows. The guide on your own API keys and cost caps goes further into where keys should be kept.
Logging, cost caps and alerts
Providers show usage per key or project, but rarely the way a business needs it: per customer, per feature or per department. A gateway logs every call with model, token counts, cost and response time, so the numbers can be broken down any way you like. On top of the logs come budget caps and rate limits per project, and alerts before a cap is reached. An automation caught in a loop that calls a model thousands of times then stops at the cap, instead of turning up on the next invoice.
Routing between models and providers
Models are replaced quickly, and the best choice for a task today may not be the best in six months. With a gateway, an application asks for an alias, such as a fast model for classification and a stronger one for drafting text, and the gateway decides which provider and model sit behind it. Switching model becomes a change in configuration rather than in code. Fallback chains pass a request on to another model or provider when the first is down or overloaded, and a response cache can return a stored answer to an identical request without a new call.
EU data residency and personal data
When prompts contain personal data or business secrets, it matters where they are processed. Several providers offer endpoints or regions in the EU, but each application has to be configured to use them, and one mistake sends the data somewhere else. A gateway can enforce the rule centrally: a project marked for the EU only reaches EU endpoints, whatever the application asks for. That does not replace the legal work under GDPR, but it turns a policy into something that is checked on every call.
When you do not need a gateway
A single application using one provider, with small volumes and a developer who keeps an eye on the invoice, does well without one. The provider's own console gives keys per project, usage figures and spending limits, and that is often enough for a pilot. A gateway starts to pay off when there are several applications or teams, more than one provider, customers whose usage has to be kept apart, or requirements on where data is processed. It is one more component to run, so it should solve a problem you have.
How ModelHarbour does it
ModelHarbour is Anno One's own AI gateway. The customer keeps the contract and the invoice with OpenAI, Anthropic or Azure, and the keys are stored encrypted and used only by a minimal relay service. It offers an OpenAI-compatible endpoint with streaming, model aliases, fallback chains and response caching. Budget caps, rate limits and a webhook warn when a budget nears its limit, alongside a hard cap set at the provider. Projects in EU mode can only use EU endpoints, and the platform itself runs in the EU. Building AI into your own processes is covered under AI and automation.