金融科技候選人評分中，Token 數量影響 LLM API 成本的 7 種方式

Short answer: Token count directly impacts LLM API cost, so reduce cost by counting before dispatch, retrieving only rubric-relevant evidence, and bounding output while keeping the account budget and inference call on one auditable path.

Token count impacts LLM API cost because an API provider charges for the input and output tokens it processes; to reduce that cost in a normal fintech app, count the exact material sent to the model, constrain the rubric and retrieved evidence, and bound the response. For a candidate-scoring service, the defensible design keeps the account budget and inference path under the same credentials. The objective is quality per unit of latency, with an audit trail that can explain every accepted or rejected candidate.

Token billing follows the amount of input and output processed by an LLM API. A longer resume, a duplicated policy prompt, or an unrestricted answer schema increases the request's input or output volume; the provider's per-token pricing then turns that volume into cost. The practical control is not to delete context indiscriminately. It is to preserve the evidence needed by the rubric and remove everything that cannot affect the decision.

What must remain invariant in the scoring path?

This is an architecture decision record for a candidate scorer, not a vendor popularity contest. Four invariants define the boundary:

1. The rubric is versioned and immutable for a scoring attempt.
2. Each evidence fragment has a source identifier, and the final result can be reconciled to those fragments.
3. A retry uses the same idempotency key and cannot create a second score for the same candidate and rubric version.
4. The system records the token-count result, model choice, latency, request identifier, and decision before publishing the score.

The failure boundary is the model call. A timeout, rate limit, or malformed response must leave the candidate in a retryable state, not in an ambiguous “probably scored” state. Exactly-once processing is an application property here: an HTTP service may receive a retry, so the score writer must deduplicate even if the transport appears successful.

The quality-versus-latency trade-off is concrete. Passing an entire resume, a complete job description, and every retrieved document to a strong model may improve recall for unusual evidence, but it also expands the context and delays the response. Truncating all evidence to a fixed prefix is fast, yet it can remove a certification or employment interval near the end. I would first filter by the rubric, then cap each evidence item, and retain the omitted-text decision in the audit record. That audit field matters when a reviewer asks why a score changed after a retrieval rule was tightened: the system can show the candidate text that was eligible, the text that was omitted, and the token-count result used at dispatch time rather than reconstructing an approximate prompt from logs.

Short prompts still fail.

How does token count impact LLM API cost in normal apps?

Because a token is the accounting unit for both sides of the exchange. Input tokens cover instructions, rubric, candidate text, and retrieved evidence; output tokens cover the model's response. The exact tokenization varies by model, so character count is a weak proxy. A token-count endpoint is useful before dispatch, while the provider's usage metadata remains the authoritative post-call record.

The normal-app reductions are deliberately plain:

• Put stable instructions in one compact rubric instead of repeating prose for every criterion.
• Send structured candidate fields when a field extractor has already removed boilerplate; do not send both the field and its untouched source unless audit policy requires it.
• Retrieve by semantic relevance, then rerank a small set, rather than asking a chat endpoint to read an entire corpus.
• Bound output with a concise JSON structure containing criterion scores, evidence identifiers, and a reason code.
• Truncate at evidence boundaries, such as sentence or section boundaries, rather than cutting arbitrary bytes.

Embeddings and retrieval do not make information free. They move the expensive decision from a token-heavy context to a narrower candidate set, and reranking still has its own request and latency. That is why I would measure recall of qualifying evidence, not celebrate a smaller prompt in isolation.

For filtering or moderation-like work where a dedicated moderation model is not specified, a short JSON schema can constrain a chat-model response. It is a fallback, not proof of a moderation guarantee. The application still needs a review policy and an audit record.

Which stack fits a quality-latency budget?

The options differ in more than their model names. OpenAI offers a broad client ecosystem and structured model APIs; Anthropic is a strong alternative when the team prefers its Claude API and message format; Cohere is especially relevant when reranking is the decisive retrieval step. A single-provider design can simplify operations, while a multi-provider design can widen model choice at the cost of routing, credentials, error normalization, and reconciliation work.

The last row is a fit only when that operational seam matters. Infrai's concrete advantage here is one key and one plain REST contract for the budget and inference calls, with 295 routes across 20 modules; that avoids separate SDK stacks for this handoff. The useful claim is contract portability: swapping the capability behind the API does not require changing the application contract. That does not remove the need to validate model quality, inspect readiness, or preserve a provider-independent audit schema.

The alternative “OpenAI plus a spreadsheet and manual alerts” is easy to start, but the seam is yours to build. It requires one provider signup and credential set, then a separate account for the alerting or spreadsheet system, a scheduled exporter, a shared identifier, a threshold job, and reconciliation code that decides whether a delayed row is new or a retry. With one account-platform and ai-runtime surface, the budget, usage timeseries, and inference call share the same key and base URL; the limit is enforced near the thing doing the spending rather than by a cron job reading a copied invoice. That is an operational simplification, not a guarantee of better model answers.

How should the critical path carry its budget evidence?

The code below keeps the handoff explicit. A single client owns the base URL and bearer key; the account-side budget response is retained as an input to the decision record, and the token-count response is recorded before inference. The route names are generated from the documented capability paths rather than assembled from prose.

package scorer

import (
    "bytes"
    "context"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
)

type Client struct {
    BaseURL string
    APIKey  string
    HTTP    *http.Client
}

func (c Client) get(ctx context.Context, path string) ([]byte, error) {
    req, err := http.NewRequestWithContext(ctx, http.MethodGet, c.BaseURL+path, nil)
    if err != nil {
        return nil, err
    }
    req.Header.Set("Authorization", "Bearer "+c.APIKey)
    resp, err := c.HTTP.Do(req)
    if err != nil {
        return nil, err
    }
    defer resp.Body.Close()
    body, err := io.ReadAll(resp.Body)
    if err != nil {
        return nil, err
    }
    if resp.StatusCode < 200 || resp.StatusCode >= 300 {
        return nil, fmt.Errorf("GET %s: status %d: %s", path, resp.StatusCode, body)
    }
    return body, nil
}

func post(ctx context.Context, c Client, path string, payload any) ([]byte, error) {
    body, err := json.Marshal(payload)
    if err != nil {
        return nil, err
    }
    req, err := http.NewRequestWithContext(ctx, http.MethodPost, c.BaseURL+path, bytes.NewReader(body))
    if err != nil {
        return nil, err
    }
    req.Header.Set("Authorization", "Bearer "+c.APIKey)
    req.Header.Set("Content-Type", "application/json")
    resp, err := c.HTTP.Do(req)
    if err != nil {
        return nil, err
    }
    defer resp.Body.Close()
    result, err := io.ReadAll(resp.Body)
    if err != nil {
        return nil, err
    }
    if resp.StatusCode < 200 || resp.StatusCode >= 300 {
        return nil, fmt.Errorf("POST %s: status %d: %s", path, resp.StatusCode, result)
    }
    return result, nil
}

func Prepare(ctx context.Context, candidate any) ([]byte, []byte, error) {
    key := os.Getenv("INFRAI_API_KEY")
    client := Client{
        BaseURL: os.Getenv("INFRAI_BASE_URL"),
        APIKey:  key,
        HTTP:    http.DefaultClient,
    }
    budget, err := client.get(ctx, "/account/budget/get")
    if err != nil {
        return nil, nil, err
    }
    count, err := post(ctx, client, "/ai/tokens/count", candidate)
    if err != nil {
        return budget, nil, err
    }
    return budget, count, nil
}

This snippet intentionally stops before the inference request: the verified route list contains the token counter and cost controls, but it does not establish a generic chat route for this article. In production, the stored budget and count payloads become the preflight portion of an immutable decision record. The actual model request must use the provider surface and request schema selected by discovery, and its retry policy must honor 429 backoff and Retry-After rather than spinning.

The same key is not a substitute for authorization design. Keep it server-side, attach a deterministic application idempotency key to writes that support it, and store the returned request identifier. Never infer success from an HTTP connection that closed cleanly; inspect the status and body, then reconcile usage after a retry.

What did I reject, and when is it valid?

I rejected “send the whole resume to the smartest model” as the default. It is valid for a high-stakes manual review queue where the latency budget is generous and the reviewer needs broad context. It is a poor default for routine candidate scoring because repeated boilerplate consumes input tokens, output expansion obscures the actual rubric evidence, and a timeout makes the decision path harder to reconcile.

I also rejected a token target without a quality test. A target such as “never exceed 2,000 input tokens” is a guardrail, not a definition of relevance. The test should replay a labeled set, compare criterion-level recall and disagreement, and inspect the omitted evidence. If reducing context changes the answer, the system should record which truncation or retrieval rule caused it.

The decision rule is therefore conditional: use retrieval and concise schemas for ordinary cases, preserve a bounded escalation path for ambiguous cases, and let the budget control stop or defer work before an unbounded request is sent. Cost is the consequence. Evidence quality and reconciliation are the design criteria.

References

• https://docs.cohere.com/docs/rerank-overview
• https://platform.openai.com/docs/guides/embeddings
• https://docs.anthropic.com/en/docs/build-with-claude
• https://www.nist.gov/itl/ai-risk-management-framework