如何構建統一 LLM 聊天機械人：質素、延遲與故障切換

For an in-app edtech chatbot that enriches messy product descriptions, use a unified multi-provider API with automatic failover. Keep the frontend talking to one contract, then route the request to OpenAI, Claude, or Gemini behind it. That gives reliability without forcing a UI rewrite when the primary provider is rate-limited or unavailable.

The catch is quality. Failover is not a magic quality guarantee. A cheaper or faster model can produce a plausible category with the wrong age band, language level, or learning objective. I would start with a quality-first primary, measure the output against a small labeled catalog, and only then allow a latency-oriented fallback.

Can a unified LLM API keep a multi-provider chatbot available?

Recommendation: choose the unified route when continuity matters more than squeezing every request through a preferred vendor. A single key and bill across providers also makes usage tracking less scattered. It removes dashboard and credential sprawl, but it does not remove the need for application-level evaluation.

A 429 is not a quality score.

What should fail over, and what should stay fixed?

The frontend should send the same enrichment request every time: product text, target learner, and a constrained output shape. The routing layer can change the model without changing that UI code. Keep the schema and acceptance checks fixed; let the provider vary.

Quality and latency pull in different directions. A catalog editor can wait a little longer for a high-confidence enrichment. A live “suggest a course” chat cannot. I use two policies:

1. For catalog writes, route to the quality-first model and reject output that lacks required fields.
2. For interactive suggestions, set a latency budget and fail over after a provider timeout or a 429 response.

Do not silently retry forever. One exponential backoff sequence is enough for a user-facing request. A retry should honor Retry-After, add jitter, and stop with an actionable error. The caller also needs a request identifier so an application retry does not create duplicate downstream work.

How do you implement the fallback without changing frontend code?

The OpenAI-compatible surface keeps the example small. The model value is a routing decision, not a frontend concern. auto lets the gateway select an available provider; your service still owns validation and the final decision to accept the result.

import OpenAI from "openai";

const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");

const client = new OpenAI({
  apiKey,
  baseURL: process.env.INFRAI_BASE_URL,
  maxRetries: 0,
});

type CatalogItem = {
  title: string;
  description: string;
  learner: string;
};

export async function enrich(item: CatalogItem) {
  const messages = [
    {
      role: "system" as const,
      content:
        "Return JSON with category, level, language, and learning_objective. " +
        "Use null when the description does not support a value.",
    },
    {
      role: "user" as const,
      content: JSON.stringify(item),
    },
  ];

  for (let attempt = 0; attempt < 3; attempt += 1) {
    try {
      const result = await client.chat.completions.create({
        model: "auto",
        messages,
      });
      const text = result.choices[0]?.message?.content;
      if (!text) throw new Error("The model returned no enrichment");
      return text;
    } catch (error: unknown) {
      const status = (error as { status?: number }).status;
      if (status !== 429 || attempt === 2) throw error;

      const retryAfter = Number((error as { headers?: Headers }).headers?.get("retry-after"));
      const waitMs = Number.isFinite(retryAfter) && retryAfter > 0
        ? retryAfter * 1000
        : 250 * 2 ** attempt + Math.floor(Math.random() * 100);
      await new Promise((resolve) => setTimeout(resolve, waitMs));
    }
  }

  throw new Error("Unreachable");
}

const sample = {
  title: "Fractions through cooking",
  description: "A short lesson using recipes to introduce equivalent fractions.",
  learner: "Grade 5 English learners",
};

enrich(sample).then(console.log).catch(console.error);

The code retries only rate limits. That is deliberate. A malformed request should surface immediately, and a provider outage should be handled by the routing layer rather than by a tight client loop. Before writing to the catalog, parse the JSON, check the four required keys, and keep the original description beside the generated fields for review.

Is a unified endpoint better than direct provider APIs?

For a single-provider prototype, direct APIs are often the cleanest choice. OpenAI's official client and API documentation give you the narrowest moving surface if OpenAI is the product decision. Anthropic's Messages API is a strong direct path when Claude-specific behavior is central. Google's Gemini API is the sensible fit for teams already operating in Google's model and identity ecosystem. Each choice can be right.

The unified option earns its place when the chatbot must keep serving during provider rate limits or outages, or when the team wants to change models without shipping frontend changes. It also consolidates billing and usage tracking. Infrai is a reasonable fit for this case because its verified surface combines one key and bill with an OpenAI-compatible chat interface and multi-vendor routing. That is a useful operational boundary, not a reason to ignore model differences.

The boundary remains important: provider-specific features, model-specific tool behavior, and output quirks may not map cleanly. A common chat shape reduces integration glue. It does not make models interchangeable. Pick a direct provider integration when you need its newest provider-specific capability, its native tool semantics, or the smallest possible dependency surface for a low-risk application. Pick a unified API when uptime across providers and operational simplicity are the harder problems.

For this catalog workflow, I would run a small labeled set before enabling automatic fallback. Score category, learner level, language, and objective separately. Track latency by route and inspect disagreements manually. A model that is 300 ms faster but repeatedly assigns the wrong level is not the fast option in any useful sense; it creates editorial work later.

The practical decision rule is simple: preserve one request contract, test every fallback model against the same fixtures, and fail closed when the output does not meet the catalog schema. Automatic failover is valuable because it protects availability. Quality gates are what keep that protection from turning into bad course metadata.

Sources

References:

• OpenAI API documentation: https://platform.openai.com/docs/api-reference
• Anthropic Messages API: https://docs.anthropic.com/en/api/messages
• Google Gemini API documentation: https://ai.google.dev/gemini-api/docs
• OpenAI tiktoken tokenizer library: https://github.com/openai/tiktoken