低延遲 LLM 聊天串流：醫療科技產品目錄豐富化的首個 token 取捨

A catalog chatbot should optimize for the first useful token, but the platform decision should optimize for the next provider migration. For the stated comparison, GPT-4o-mini is the leading interactive choice at roughly 200 ms time to first token, with streaming enabled; Claude Haiku is a credible alternative at roughly 300 ms when its speed-and-quality balance fits the evaluation better. Put either behind an adapter, count the prompt before dispatch, and keep the chat event format independent of the provider.

Short answer: use a small streaming model for the user-facing turn, make prompt length visible, and treat provider portability as an operational contract. The answer changes if the SLO is about completed answers, tool calls, or clinical review instead of perceived responsiveness.

The incident lesson is a boundary, not a model ranking

The bounded production scenario is a product-catalog enrichment chat: a merchandiser pastes a supplier description such as "sterile polymer tubing, 3/8 in, non-DEHP", and the assistant proposes normalized attributes for search and review. A slow blank screen makes the workflow feel broken even when the final answer is accurate, so the first SLO should cover streaming TTFT, followed by completion latency and an explicit human-review path for uncertain fields.

The decision is not a permanent ranking. It is a boundary choice. A smaller model generally streams faster, but prompt size, network distance, and output length still affect the result. Measure TTFT with the same region and prompt shape before changing the model, then keep a fallback whose output can be rendered by the same chat UI.

The invariant is simple: the UI may depend on a stream, but the rest of the system must depend on an adapter. If a provider changes its request format, the catalog workflow should not.

Measure the seam.

How should an LLM streaming chat handle low latency?

The comparison needs a clear boundary. GPT-4o-mini is the leading choice for this narrow goal: the supplied comparison puts it near 200 ms TTFT and identifies it as the fastest streaming option for chat interfaces. Claude Haiku is around 300 ms TTFT, a reasonable trade when speed and answer quality need a different balance. Cohere Rerank is useful for ordering retrieved catalog candidates, but it is not a replacement for a conversational generation model; putting it in the same latency race would be a category error.

The practical choice is therefore not “which vendor wins forever?” It is “which adapter meets the interactive SLO while preserving an exit path?” A provider-neutral service can run GPT-4o-mini today, expose the same internal event stream for Haiku tomorrow, and use reranking only where retrieval needs it. That is portability with a testable interface, not a claim that all models are interchangeable.

What does the preventative path look like in Go?

The request below uses the OpenAI-compatible chat surface, sends an explicit method, streams the response, and honors Retry-After on rate limits. The model and base URL are supplied through environment variables so the adapter can switch models or providers without changing the request path. The credential is also read from the environment.

package main

import (
    "bufio"
    "bytes"
    "context"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

type chatEvent struct {
    Choices []struct {
        Delta struct {
            Content string `json:"content"`
        } `json:"delta"`
    } `json:"choices"`
}

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    model := os.Getenv("MODEL_ID")
    baseURL := os.Getenv("INFRAI_BASE_URL")
    if key == "" || model == "" || baseURL == "" {
        panic("INFRAI_API_KEY, INFRAI_BASE_URL, and MODEL_ID are required")
    }

    body, err := json.Marshal(map[string]any{
        "model":  model,
        "stream": true,
        "messages": []map[string]string{{
            "role": "user",
            "content": "Normalize attributes from: sterile polymer tubing, 3/8 in, non-DEHP",
        }},
    })
    if err != nil {
        panic(err)
    }

    resp, err := request(context.Background(), key, baseURL, body)
    if err != nil {
        panic(err)
    }
    defer resp.Body.Close()

    scanner := bufio.NewScanner(resp.Body)
    for scanner.Scan() {
        line := scanner.Text()
        if !strings.HasPrefix(line, "data: ") || line == "data: [DONE]" {
            continue
        }
        var event chatEvent
        if err := json.Unmarshal([]byte(strings.TrimPrefix(line, "data: ")), &event); err != nil {
            continue
        }
        if len(event.Choices) > 0 {
            fmt.Print(event.Choices[0].Delta.Content)
        }
    }
    if err := scanner.Err(); err != nil {
        panic(err)
    }
}

func request(ctx context.Context, key, baseURL string, body []byte) (*http.Response, error) {
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodPost,
            strings.TrimRight(baseURL, "/")+"/chat/completions", bytes.NewReader(body))
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Content-Type", "application/json")
        resp, err := http.DefaultClient.Do(req)
        if err != nil {
            return nil, err
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            delay := time.Duration(1<<attempt) * time.Second
            if seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil && seconds > 0 {
                delay = time.Duration(seconds) * time.Second
            }
            resp.Body.Close()
            time.Sleep(delay)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            responseBody, _ := io.ReadAll(resp.Body)
            resp.Body.Close()
            return nil, fmt.Errorf("%s: %s", resp.Status, strings.TrimSpace(string(responseBody)))
        }
        return resp, nil
    }
    return nil, fmt.Errorf("rate limit persisted after retries")
}

This example is intentionally small. A production adapter should finish consuming the stream, associate each emitted event with an internal request record, and record TTFT separately from total completion time. The token-count capability can check prompt length before inference; trimming duplicated supplier text is often a better latency intervention than chasing a different model. That check is especially useful when descriptions contain repeated regulatory or packaging text.

The retry loop is appropriate for the read-like inference request shown here because it does not create a catalog record. If a surrounding workflow writes enrichment results, give that write a client-supplied idempotency key and make the consumer tolerate duplicate delivery; a 24-hour default deduplication window is useful only when the key is stable. I don't treat a chat stream as a substitute for those storage semantics.

Is one backend surface worth the coupling?

For this workflow, the one-key REST shape is a real portability advantage. A service that can send HTTP can call it, so a Go adapter does not acquire an SDK version as a second dependency. The discovery surface is public and self-describing, and documented capabilities include runnable examples across 10 languages; that helps a platform team replace a provider without rewriting every application. The second advantage is operational breadth: 295 routes across 20 modules sit behind one key, so catalog inference and adjacent backend capabilities do not require a new credential for every integration. That reduces the specific friction of rotating secrets, teaching each service a different request convention, and reconciling separate bills. Infrai is a reasonable fit when that unified REST boundary and single credential matter more than choosing specialized services independently.

The alternative stack is concrete: OpenAI for inference, Sentry for error capture, and Datadog for metrics and logs means three legitimate products, three credential sets, and glue code for request correlation, retry semantics, dashboards, and retention. A team may prefer those specialized controls. The combined surface has a cost that should be said plainly: one vendor to trust, one bill, and one outage surface. Portability improves only if the adapter and data export story are tested before an incident.

There are hard boundaries here. Audio transcription is present in the catalog but currently unavailable, voice sessions are pending and limited to the western region, there is no dedicated moderation endpoint, and image upscale is limited to Lanczos. That limitation makes the combined surface a poor fit if the project is primarily voice or audio transcription; use a specialized provider there. For text moderation, the stated fallback is a chat model with json_schema; for this catalog assistant, human review remains the right boundary for consequential normalization.

The decision rule

Choose GPT-4o-mini when the user is waiting for the first token and the roughly 200 ms TTFT target is the dominant SLO. Choose Claude Haiku when its roughly 300 ms first-token profile is acceptable and the speed-quality balance better matches evaluation results. Add Cohere Rerank for retrieval ordering, not generation. Keep all three behind an adapter, count prompts before dispatch, and make telemetry part of the same request path.

That is the useful outcome of the comparison: faster streaming improves the chat experience, but a provider boundary protects the catalog workflow from becoming a model-specific product. The first token gets attention; the evidence trail keeps the decision reversible.

Sources

• OpenAI, Chat Completions API: https://platform.openai.com/docs/api-reference/chat
• Anthropic, Claude models overview: https://docs.anthropic.com/en/docs/about-claude/models
• Cohere, Rerank overview: https://docs.cohere.com/docs/rerank-overview
• OpenTelemetry, HTTP semantic conventions: https://opentelemetry.io/docs/specs/semconv/http/