Python 中 LLM API token 數量對 CRM 成本及用量削減的影響

The operational constraint is easy to miss: token count has a direct impact on LLM API cost when a normal app sends a full sales-call transcript through a model repeatedly. Short answer: reduce token usage effectively by keeping the CRM output contract small, selecting evidence before generation, and recording input and output counts in the evaluation record. Do that before choosing a provider.

The goal is not a shorter transcript at any cost. The goal is the smallest evidence window that still preserves the customer need, owner, commitment, next action, and unresolved question. A compact JSON result helps too, because CRM code rarely needs a paragraph explaining why the action was chosen.

Measure it first.

In a normal app, the cheapest-looking prompt is not automatically the best prompt.

How does token count impact LLM API cost in a CRM app?

An LLM request has input and output volume. The transcript, system instruction, retrieved passages, and schema are inputs; the extracted CRM actions are output. Providers charge for processed tokens, so both sides affect the bill. A retry that resends a long transcript can multiply the waste even when the final response is only a few fields.

I start with a notebook-shaped baseline: pass the entire call to a model, ask for a summary, and inspect the result. It is a useful proof of concept. It is a poor production policy. Greetings, repeated objections, speaker labels, and side conversations occupy context without necessarily supporting a CRM action.

The correction is an evaluation boundary. Store the original call, the selected evidence, the pre-request token count, the post-request output count, and the normalized action object together. Then score the result against a fixed rubric. A lower count is not a pass if the model drops the promised follow-up date. In my Python workers, I would keep the 12,000-character guard visible in the record, label it as a pre-count limit, and let the target provider's tokenizer supply the actual billing count.

One missing owner can cost more operationally than several extra lines of context.

Can retrieval reduce tokens without losing the action?

Yes, when retrieval is tied to the fields the application must fill. I would create separate queries for the customer need, commitments, ownership, and open questions, then merge a bounded set of relevant passages. Embeddings are useful for semantic search because the chat request receives evidence instead of the entire call. A reranker can refine that shortlist when similarity returns several near-duplicates; Cohere's reranking documentation describes this second-stage pattern.

Retrieval creates a new failure mode. The passage with the highest semantic score may mention a product objection while the next passage contains the actual owner and date. That is why the eval set needs hard examples: long calls, repeated names, corrections, and commitments made near the end. Record retrieval recall and truncation events beside token counts.

Here is a deliberately small preprocessing policy. It is not a tokenizer, and its character limit is not a billing number. It is a guardrail that keeps an accidental unbounded transcript out of the counting step.

from dataclasses import dataclass
from typing import Iterable


@dataclass(frozen=True)
class Evidence:
    text: str
    relevance: float


def select_evidence(
    items: Iterable[Evidence],
    max_characters: int = 12000,
) -> tuple[str, bool]:
    selected: list[str] = []
    used = 0
    truncated = False

    for item in sorted(items, key=lambda value: value.relevance, reverse=True):
        text = item.text.strip()
        if not text:
            continue
        remaining = max_characters - used
        if remaining <= 0:
            truncated = True
            break
        if len(text) > remaining:
            selected.append(text[:remaining])
            truncated = True
            break
        selected.append(text)
        used += len(text)

    return "\n\n".join(selected), truncated


if __name__ == "__main__":
    evidence = [
        Evidence("The customer needs a renewal proposal by Friday.", 0.91),
        Evidence("The account owner will send the proposal.", 0.88),
        Evidence("The call opened with unrelated small talk.", 0.12),
    ]
    context, was_truncated = select_evidence(evidence)
    print({"context": context, "truncated": was_truncated})

For the generation step, the same contract can sit behind an OpenAI-compatible Infrai client. Its public discovery surface is self-describing, and the platform exposes one REST API across 295 routes in 20 modules; that makes the adapter boundary easier to inspect, but it does not remove the need to count tokens.

import os
import time

from openai import APIError, OpenAI, RateLimitError


context = os.environ["CRM_CONTEXT"]
client = OpenAI(
    api_key=os.environ["INFRAI_API_KEY"],
    base_url=os.environ["INFRAI_BASE_URL"],
)

messages = [
    {
        "role": "system",
        "content": (
            "Extract CRM actions. Return only customer_need, owner, "
            "next_action, and open_questions as JSON."
        ),
    },
    {"role": "user", "content": context},
]

for attempt in range(3):
    try:
        response = client.chat.completions.create(
            model="deepseek-v4.1-flash",
            messages=messages,
        )
        break
    except RateLimitError as error:
        if attempt == 2:
            raise
        retry_after = getattr(error.response, "headers", {}).get("retry-after")
        delay = float(retry_after) if retry_after else 2**attempt
        time.sleep(delay)
    except APIError as error:
        raise RuntimeError(f"LLM request failed: {error}") from error
else:
    raise RuntimeError("The LLM request did not complete")

print(response.choices[0].message.content)

The API key comes from the environment, and the SDK surfaces non-success responses as errors instead of pretending every request succeeded. Keep the count and response metadata with this call in the eval record. If the provider adapter changes, the CRM fields and test cases should not.

After this selection, the application should use the target provider's counting facility or tokenizer to count the serialized request. The character guard prevents runaway input; it does not claim to predict tokens. On the output side, request only the fields that the CRM worker consumes. A JSON schema for filtering or classification can be more economical than asking for a polished explanation.

The tradeoff is recall. A smaller context can hide a correction, a due date, or a sentence that changes who owns the task. Keep the original transcript for replay, but send only the evaluated evidence window to generation. That gives the team a way to inspect a miss instead of guessing why a short answer was wrong.

How should Python teams compare providers for this pipeline?

Portability should be tested at the application contract, not inferred from similar-looking chat calls. OpenAI, Anthropic, and Google Gemini are real alternatives, but their request formats, model identifiers, token accounting, and response handling remain provider concerns. The CRM worker should not know those details.

The last row is not a quality shortcut. A compatibility layer can keep the code that turns an extracted action into a CRM update stable while the provider-specific request stays in an adapter. The stable contract is the benefit: the thing behind it can move without rewriting the workflow. The layer still has to expose enough token and error metadata for the eval harness to catch regressions.

I would keep the adapter thin. It owns authentication, model names, message formatting, provider token counting, and provider errors. The application owns customer_need, owner, next_action, and open_questions. If a provider switch changes those business fields, the boundary is in the wrong place. If the boundary hides input and output counts, it is too opaque to govern cost.

There is a limit. Choose a direct provider integration when a provider-specific feature is the main product requirement or when the team has no need to move the backend. Choose the compatibility boundary when several backends must satisfy the same CRM contract and the evaluation suite can compare them fairly.

What should the eval gate record before shipping?

Use the same call set, prompt intent, output fields, and scoring rubric for every candidate. I would record action accuracy, owner accuracy, unresolved-question recall, malformed JSON, input tokens, output tokens, retrieval misses, truncation, and retries. The point is not to produce a pretty cost dashboard. It is to see which token reduction changes the answer.

The first experiment can compare two paths:

1. Full transcript plus a broad instruction.
2. Retrieved evidence plus a compact output contract.

Keep the same cases and inspect both false omissions and extra context. If Path 2 reduces tokens but loses commitments, increase the evidence window or improve retrieval. If it preserves the rubric, promote the policy into the worker and keep a replay sample for later prompt changes.

Token counting belongs before the request and after the response. Preflight catches an input that crossed its intended budget. Postflight catches a model that returned an essay after being asked for fields. Both measurements matter because a short input can still produce a long answer.

Do not use a character limit as a substitute for token accounting, and do not treat embeddings as automatically cheaper. Retrieval calls have their own cost and latency, and a bad query can discard the sentence that carries the owner. The eval record is where that tradeoff becomes visible.

The decision rule

Start with the smallest CRM schema that operations can actually use. Retrieve only evidence that supports those fields, count the serialized input and output, and compare OpenAI, Anthropic, Gemini, or a compatibility layer on the same cases. Keep the transcript for replay and the evidence window for generation.

The winning design is not the one with the shortest prompt. It is the one that stays within an inspectable token budget while preserving action accuracy across representative sales calls. That is the boundary worth carrying from a notebook into production.

Sources

• https://platform.openai.com/docs/guides/embeddings
• https://platform.openai.com/docs/guides/text-generation
• https://docs.anthropic.com/en/docs/build-with-claude/token-counting
• https://ai.google.dev/gemini-api/docs/tokens
• https://docs.cohere.com/docs/rerank-overview