企業級 LLM 閘道參考架構

A single developer calling a model through a gateway needs little more than a base URL and a key. An enterprise running hundreds of AI features across many teams, regions, and regulatory regimes needs something more carefully structured. This article lays out a reference architecture for deploying an LLM gateway at enterprise scale, walking through the split between control and data planes, regional deployment, authentication and access control, budgets, guardrails, observability, and the deployment topologies that determine where your data lives. TrueFoundry is used throughout as the concrete example, shown to make the architecture tangible rather than to make a pitch.

The Organizing Principle: Split the Control Plane from the Data Plane

The most important architectural decision in an enterprise gateway is to separate the control plane from the data plane. These are two distinct components with two distinct jobs, and keeping them apart is what lets the system be both centrally governed and fast at the edge.

The control plane is the centralized brain. It holds configuration: which models exist, how routing is defined, who has access to what, what budgets and rate limits apply, which guardrails are active. It is where administrators and platform teams make decisions. It is not in the path of any model request.

The data plane, also called the gateway plane, is the only component that sits in the request path. When an application calls the gateway, it is the data plane that receives the request, applies routing, meters tokens, enforces limits, and forwards to the provider. Crucially, the data plane does not phone home to the control plane on every request. It reads its configuration and cache from memory, so the per-request work stays local and fast. This is the architectural reason a gateway can add typical latency overhead under 5 milliseconds, internally around 3 to 4 milliseconds, at throughput on the order of 200 to 250 requests per second per single CPU: the in-path component is lean, and the heavy coordination happens out of band.

Regional Data Planes and Data Residency

Because the data plane is the only thing in the request path and reads its configuration from memory, you can deploy it regionally. An enterprise with users and workloads in several regions can run a data plane close to each one, so requests do not have to cross the globe to reach a gateway. Each regional data plane pulls the same centrally managed configuration from the control plane, which means policy is consistent everywhere while traffic stays local.

This regional model is also the foundation of data residency control. Where the data plane runs, and where it is allowed to send request and response data, is something the architecture lets you constrain. That matters enormously for organizations bound by rules about where personal or regulated data may be processed and stored.

A Request's Journey Through The Architecture

It helps to trace a single request end to end. An application, authenticated with its credential, sends an OpenAI-compatible chat-completions call to the regional data plane nearest it. The data plane validates the credential and checks the caller's access rights and rate limits, drawing on configuration it already holds in memory. It resolves the requested model or virtual model group, applies the routing policy to select a target, and if guardrails are configured, inspects the prompt before it leaves. It forwards the request to the chosen provider, streaming the response back as it arrives, and as the stream flows it accumulates token usage. On completion it records that usage against the relevant team, application, and user for budget attribution and observability, applies any response-side guardrails, and returns the result. If the chosen target had failed, the data plane would have transparently retried against a fallback candidate. Throughout, the control plane was never in the path; it had already supplied the configuration that made every one of these decisions possible.

Authentication and Role-based Access Control

At enterprise scale, credentials are not an afterthought. The architecture distinguishes two credential types by purpose. A Personal Access Token (PAT) represents a human and suits development and interactive use. A Virtual Account Token (VAT) represents an application or service and suits production workloads. Keeping these distinct means a departing employee's personal token can be revoked without taking down production services, and a compromised application token can be rotated without disturbing developers.

Around those credentials sits role-based access control. Access to models, the ability to change routing, the authority to set budgets, the visibility of usage data: all of these can be granted by role rather than handed out uniformly. A platform team might administer configuration while individual product teams can only call the models they have been granted and view their own usage. This is how a single shared gateway serves many teams without any team being able to affect another.

Budgets, Rate Limits, and Guardrails as Policy

Because every request flows through the data plane and its usage is metered in tokens, the control plane can express spend and protection as policy. Budgets can be set per team, per application, or per user, and enforced so that once exhausted, further requests are refused rather than silently accruing cost. Rate limits cap request or token throughput per consumer, protecting shared provider quotas from any single noisy service. Guardrails, applied in the data plane where the prompt and response text is visible, screen content on the way in and on the way out. The important architectural point is that all three are defined centrally in the control plane and enforced locally in each data plane, so a policy written once applies everywhere the gateway runs.

Observability

Centralizing traffic through the data plane makes the whole fleet observable from one place. Request volumes, latencies, error rates, token consumption, and cost can be broken down by model, team, and application. For an enterprise this is both an operational necessity, letting platform teams spot a failing provider or a runaway workload, and a governance necessity, letting finance and compliance answer who spent what and which models handled which traffic.

Deployment Topologies and Where Data Lives

The same architecture supports several deployment topologies, and the choice among them is largely a choice about where your request and response data resides. There are four to consider:

• SaaS. The gateway is operated for you, the simplest option to run, with the base URL on SaaS being https://gateway.truefoundry.ai. -** Hybrid.** The control plane is managed for you, but request and response data is kept in your own object storage, so the sensitive payloads never leave infrastructure you control while you still get centralized management.
• Self-hosted. You run the components in your own environment end to end.
• Air-gapped. For the most restrictive environments, the system runs with no external connectivity at all.

These topologies exist precisely because enterprises differ in how much they must control where data flows. A team with light requirements may happily use SaaS; a bank or a defense contractor may need self-hosted or air-gapped. On the compliance side, the platform supports SOC 2, HIPAA, GDPR, and ITAR, which maps onto the range of regulated industries these topologies are meant to serve.

A Minimal Configuration to Anchor the Picture

Even in an enterprise deployment, an application's view of the gateway stays simple, which is the point of putting all the complexity in the control plane. An application still just points the standard OpenAI client at its regional data plane endpoint and supplies its production credential. Using environment variables, that looks like this:

Application-side configuration. All routing, budgets,
access control and guardrails are defined centrally in
the control plane, not here.

export OPENAI_BASE_URL="https://gateway.truefoundry.ai"
export OPENAI_API_KEY="your_virtual_account_token"

Everything that makes the deployment enterprise-grade, the regional data planes, the RBAC, the budgets, the guardrails, the residency rules, is configured once in the control plane and enforced in each data plane. The application developer sees none of it; they see a base URL and a key. That contrast is the whole design in miniature: complexity concentrated where it can be governed, simplicity preserved where work gets done.

Putting The Pieces Together

A sound enterprise reference architecture for an LLM gateway therefore reads as a set of deliberate separations. Configuration is separated from the request path by splitting the control plane from the data plane. Latency is minimized by letting regional data planes serve requests from in-memory configuration. Governance is centralized through RBAC, budgets, rate limits, and guardrails defined once and enforced everywhere. Data residency is honored by choosing a deployment topology, from SaaS through hybrid and self-hosted to air-gapped, that places request and response data where regulation requires. And observability ties it together by making the whole fleet legible from a single vantage point. Each separation buys a specific property, and together they let one gateway serve an entire enterprise without forcing any team to trade speed for control.

Key Takeaways

• Separate the control plane, which holds configuration and routing centrally and stays out of the request path, from the data plane, the only in-path component, which reads config from memory for low latency.
• Deploy data planes regionally so requests stay local and data residency can be controlled, while all regions share one centrally managed configuration.
• Use distinct credentials for people (PAT) and applications (VAT), and layer role-based access control so one shared gateway safely serves many teams.
• Define budgets, rate limits, and prompt and response guardrails as central policy, enforced locally in every data plane.
• Choose a deployment topology, SaaS, hybrid, self-hosted, or air-gapped, to place request and response data where compliance regimes such as SOC 2, HIPAA, GDPR, and ITAR require.