Perplexity 拆解 LLM gateway 運作機制與生產環境風險

Perplexity published a practical guide to LLM gateways on October 27, 2026, walking engineering teams through how the intermediary layer routes, authenticates, caches, and logs model calls and cataloging the security and reliability risks of concentrating provider credentials in one service.

Gateway Definition and Taxonomy

The guide defines an LLM gateway as a reverse proxy for model calls: applications send requests to the gateway instead of directly to providers such as OpenAI, Anthropic, or Google, and the gateway forwards those requests while applying the operator’s rules. It presents the category as an answer to sprawl, opening with a September 2026 post on Reddit’s r/LLMDevs in which an engineer described a workplace where everyone held their own keys for Claude, OpenAI, AWS Bedrock, and other inference providers.

The guide’s comparison table separates four gateway types. An API gateway sits in front of backend services and decides whether a client is authenticated and under its rate limit. An LLM gateway sits in front of model providers and decides which model gets a request, who pays for it, and whether it came back. An MCP gateway decides whether an agent may call a given tool with given arguments. The guide uses “AI gateway” for vendor products that bundle some combination of those roles.

The Six-Step Request Lifecycle

The guide explains the mechanics by tracing one request from application to model and back. First, the application sends a chat request, often in an OpenAI-compatible format, to the gateway’s endpoint. The gateway validates the credentials, often a virtual key issued to a team or product, and, when input guardrails are wired in, can scan prompts for sensitive data or attempts to override the model’s instructions. If caching is enabled, an eligible matching request can be answered from a saved response, and some gateways offer semantic caching that matches requests judged similar rather than identical. The gateway then looks up the matching route in its configuration and can fall back to a backup provider if the first provider times out. It converts the request into the chosen provider’s format, authenticates the outgoing call, and converts the response back for the application. With logging enabled, it records who made the call, which model served it, token counts, latency, and cost.

Security and Reliability Risks

The guide’s central warning is that the gateway concentrates credentials. Microsoft’s security team has listed what a LiteLLM deployment may hold or retrieve: provider API keys, the LiteLLM master key, virtual-key records, database connection strings, routing configuration, and tenant policy data. In that August 26, 2026 post, Microsoft described an observed LiteLLM gateway compromise in which attackers harvested credentials from the gateway runtime.

On October 13, 2026, Check Point Research disclosed two high-risk vulnerabilities in LiteLLM: CVE-2026-35029, an access-control flaw that let authenticated users, including read-only ones, reach administrative functions and take over the gateway, and CVE-2026-59823, a server-side request forgery flaw that let users redirect gateway requests to an attacker-controlled server and capture AI provider API keys. Both are fixed in LiteLLM version 1.83.9. Check Point reported LiteLLM at 97 million downloads per month and more than 59,000 GitHub stars, and said its coordinated disclosure to LiteLLM began in October 2025.

The guide also tells teams to set a failure policy for guardrails. It describes hooks as points in the request path where the gateway hands content to an external scanner for a verdict, noting that Portkey’s documentation names before_request_hooks and after_request_hooks and that TrueFoundry runs four hooks, adding checks before and after MCP tool calls.

On reliability, the guide points to documented outages. Cloudflare’s own account of its June 12, 2025 outage states that its AI Gateway saw error rates peak at 97% of requests because AI Gateway relies on Workers KV for client and internal configurations; the overall outage lasted 2 hours and 28 minutes. OpenRouter’s postmortem of its February 17 and 19, 2026 outages states the incidents lasted 38 and 35 minutes after a third-party caching dependency failed, produced 401 “User not found” errors, and led to remediations including circuit breakers and returning 503 errors instead of 401 when infrastructure problems block authorization lookups.

Deployment Options, Pricing, and Vendor Specifics

The guide lays out four deployment paths. Teams can write a minimal gateway in-house — accepting requests, checking the caller’s key, adding the provider credential, forwarding the call, and logging the result — or self-host an open-source gateway, with LiteLLM, Bifrost, and Helicone named as options. Managed gateways named include Cloudflare AI Gateway, Vercel AI Gateway, OpenRouter, and Portkey’s hosted gateway. A fourth path extends an API gateway a company already runs, naming Kong AI Gateway, Apigee’s AI gateway policies, and Azure API Management’s AI gateway.

On cost, the guide states that LLM gateways range from free open-source software to paid plans starting around $50 a month with custom enterprise pricing, and that some charge a percentage instead: OpenRouter’s Standard plan, it notes, adds 5.5% to credit purchases.

The guide closes with vendor-specific answers. It states that Amazon Bedrock AgentCore Gateway supports model calls through what AWS calls “inference targets,” letting applications use one endpoint to reach providers including Amazon Bedrock, OpenAI, and Anthropic. It notes that Perplexity’s own Agent API supports OpenAI’s Responses API interface, so teams can reuse existing OpenAI client libraries, and that the Agent API lets users list up to five models, including models from different providers, in preferred order. The guide also names Perplexity’s open-source Numbat, which connects to supported agents through local hooks and can inspect planned actions such as shell commands or file writes before they run, with blocking optional and off by default.