如何比較 LLM 聊天機械人的每 token 生產成本，並設定三類失敗預算

How to Compare LLM Chatbot Production Cost Per Token — With 3 Failure Budgets

Short answer: for an in-app LLM chatbot, compare production cost per accepted decision, not the advertised cost per token alone. In an edtech candidate-scoring workflow, count tokens plus retries, retrieval, timeout handling, and human review. Set three budgets before selecting a model or service: malformed-output budget, latency budget, and spend budget. A cheaper token can lose quickly if it produces more unusable decisions.

The useful unit is an accepted score, not a request.

That distinction matters when the response must contain fixed fields such as score, evidence, and missing_requirements, rather than a paragraph that a person can interpret generously.

This method has limits: it cannot predict quality from token volume, and it cannot replace a representative evaluation corpus. The tradeoff is deliberate: a small, auditable contract gives safer operations, while a richer contract can preserve more nuance but raises validation and review work. This approach is unsuitable for an open-ended tutoring conversation where a human can accept a useful answer without fixed fields. It is a comparison discipline, not a promise that one runtime will fit every workload.

Measure it.

How should an LLM chatbot measure production cost per accepted token?

A per-token table answers only one narrow question. It does not tell you how many tokens a candidate record needs, how often the response must be retried, or what happens when the output cannot be parsed. A production ledger should record at least these values for each request:

Use this calculation for a comparable sample:

type Run struct {
    InputTokens       int
    OutputTokens      int
    RetryInputTokens  int
    RetryOutputTokens int
    Accepted          bool
    NeedsReview       bool
}

func acceptedResultRate(runs []Run) float64 {
    if len(runs) == 0 {
        return 0
    }
    accepted := 0
    for _, run := range runs {
        if run.Accepted {
            accepted++
        }
    }
    return float64(accepted) / float64(len(runs))
}

func effectiveWork(runs []Run) int {
    total := 0
    for _, run := range runs {
        total += run.InputTokens + run.OutputTokens
        total += run.RetryInputTokens + run.RetryOutputTokens
    }
    return total
}

The code does not assign a score and should not. Scoring policy belongs in a versioned rubric and a separate evaluator; this ledger only makes the runtime comparison honest. I initially expected output tokens to dominate because the score response is short. The larger surprise in this design is that a malformed short response still consumes a full request and forces another decision path.

Step 1: Define the acceptance contract before the call

A candidate decision is accepted only when it is parseable, complete, and consistent with the rubric version attached to the request. Treat those as separate checks. A response can be valid JSON and still omit a required explanation, use an out-of-range score, or cite a requirement that is not present in the supplied evidence.

Keep the contract small. For example, the application can require an integer score from 0 through 100, a rubric identifier, and an array of evidence references. The application should reject extra ambiguity at the boundary instead of asking downstream code to guess what the model meant. JSON Schema is a suitable standards-based description for that boundary; its specification defines the vocabulary and processing model, while the runtime remains responsible for applying the result to its own policy.

type Decision struct {
    RubricVersion string   `json:"rubric_version"`
    Score         int      `json:"score"`
    Evidence      []string `json:"evidence"`
    NeedsReview   bool     `json:"needs_review"`
}

func validateDecision(d Decision, expectedRubric string) error {
    if d.RubricVersion != expectedRubric {
        return fmt.Errorf("rubric version mismatch")
    }
    if d.Score < 0 || d.Score > 100 {
        return fmt.Errorf("score outside permitted range")
    }
    if len(d.Evidence) == 0 {
        return fmt.Errorf("decision has no evidence")
    }
    return nil
}

In real code, import fmt and decode with the standard JSON package before validation. The important boundary is the order: decode, validate shape, validate rubric semantics, then persist or display. Do not let a failed validation silently become a zero score. That is a data-quality defect disguised as a successful request.

Step 2: Spend retries as a bounded resource

Retries need a reason, a limit, and a distinct outcome. Retry a transient transport failure when the request can safely be repeated. A schema failure may justify one constrained repair attempt, but repeated repair is not a quality strategy; it is an unpriced loop. Never retry a rejected decision indefinitely just because the user is waiting in the chat interface.

A practical policy for a first rollout is one repair attempt, an explicit needs_review result after the second rejection, and a request identifier that ties both attempts to one candidate and rubric version. The number is a policy choice, not a universal benchmark; record it so the comparison remains reproducible.

type Outcome string

const (
    Accepted    Outcome = "accepted"
    NeedsReview Outcome = "needs_review"
    Failed      Outcome = "failed"
)

func classify(attempts int, valid bool, transportErr error) Outcome {
    if transportErr != nil && attempts < 2 {
        return Failed // the caller may retry a transport failure once
    }
    if valid {
        return Accepted
    }
    if attempts >= 2 {
        return NeedsReview
    }
    return Failed
}

The caller should attach reason codes such as transport, decode, schema, or rubric. Those labels let on-call staff tell a capacity problem from a contract problem without reading every transcript. Log token counts and latency, but redact candidate text and other sensitive education data; usage telemetry does not require storing the underlying application content.

Step 3: Compare buy and build on the same evidence

The buy-versus-build choice is an operational decision, not a referendum on which API has the smallest published number. A managed runtime can reduce maintenance for quotas, transport, and model serving, while a self-hosted path can offer more control over data handling and deployment. Both still need a schema contract, evaluation set, timeout, and rollback path.

Run the same anonymized, consented evaluation set through each candidate path. Include short profiles, long profiles, missing fields, conflicting evidence, and deliberately adversarial formatting. Compare accepted-result rate and p95 latency alongside tokens. A 5% retry rate is not interchangeable with a 5% human-review rate: one consumes runtime budget, the other changes the service promise and staffing model.

For an edtech system, capacity planning should use peak submissions rather than a daily average. Estimate concurrent requests, maximum input size, output ceiling, retry allowance, and timeout. Then reserve headroom for rubric releases and admission bursts. The exact headroom is an internal SLO decision; document it instead of presenting it as a fact about a provider.

How do you verify a change and roll it back?

Verification begins with a fixed test corpus and a versioned rubric. Before rollout, assert that every accepted result is parseable, within range, tied to the expected rubric, and backed by evidence. Track invalid-output rate, accepted-result rate, p95 latency, token usage per accepted result, and review rate by rubric version. These are release signals, not decorative dashboards.

Canary the new runtime or prompt behind a routing flag. Keep the old path available until the new path meets the same acceptance and latency SLOs over a representative window. Roll back on a contract breach, a sustained SLO violation, a sudden rise in review outcomes, or a budget alarm. Preserve the request and rubric identifiers so a rejected candidate can be reprocessed deterministically after the cause is understood.

The rollback target should be a known-good configuration, not an improvised prompt edit. Freeze the evaluation corpus during the comparison, record the exact schema and rubric versions, and review the failed examples after traffic is safe. This is slower than celebrating a lower token line item, but it protects the decision that the application actually sells.

The final decision rule is straightforward: select the path that meets the structured-output SLO and latency SLO within the spend budget, with bounded retries and an explicit review outcome. If two paths meet those constraints, choose based on on-call ownership, data handling, portability, and migration effort. Token price is a measurement in that decision, not the decision itself.

References

• https://json-schema.org/specification
• https://platform.openai.com/docs/guides/embeddings
• https://sre.google/sre-book/service-level-objectives/