聊天機械人每次重送完整對話的成本與兩種精簡方法實測

Chat APIs are stateless. To answer message 41, your app sends messages 1 to 40 again, and you pay for all of them as input tokens, every single time.

On our own API traffic over the last 7 days (about 411,000 requests), 99.3% of all tokens processed were input tokens. The median request sent 3,782 tokens in and got 147 back. Most LLM bills aren't about long answers. They're about long histories.

Here's how much that costs on a realistic chat, and two simple fixes, measured on real API calls.

The test

A support bot after 40 turns, then one more question. Three strategies:

1. Full history: what most tutorials do
2. Last 6 turns: a sliding window
3. Summary + last 4 turns: older turns compressed into a 60-word note by a cheap model

"""trim_context.py: what does chat history really cost?"""
import os
from openai import OpenAI

client = OpenAI(base_url="https://api.qubax.ai/v1",
                api_key=os.environ["QUBAX_API_KEY"])
MODEL, PRICE_IN, PRICE_OUT = "claude-sonnet-5", 0.3835, 1.5339  # $/1M

history = [{"role": "system",
            "content": "You are a concise support agent for an invoicing SaaS."}]
for i in range(40):
    history += [
        {"role": "user", "content": f"Question {i}: how do I change the due "
         f"date on invoice #{1000+i}? I also need to know whether the "
         "customer gets notified and if late fees recalculate."},
        {"role": "assistant", "content": f"Open invoice #{1000+i}, click "
         "Edit, change the due date and save. The customer gets an update "
         "email, and late fees recalculate from the new date."},
    ]
question = {"role": "user", "content": "Last one: can I bulk-change due "
            "dates for all invoices of one customer?"}

def ask(messages):
    r = client.chat.completions.create(model=MODEL, messages=messages,
                                       max_tokens=120, temperature=0)
    return r.choices[0].message.content, r.usage

def keep_last(msgs, turns):
    return msgs[:1] + msgs[1:][-2 * turns:]

def summarize_old(msgs, keep_turns):
    old, recent = msgs[1:][:-2 * keep_turns], msgs[1:][-2 * keep_turns:]
    note = client.chat.completions.create(
        model="glm-5.3-flash", temperature=0, max_tokens=800,
        messages=[{"role": "user", "content":
                   "Summarize this support chat in at most 60 words. "
                   "Keep names, IDs and terms.\n\n"
                   + "\n".join(f"{m['role']}: {m['content']}" for m in old)}],
    ).choices[0].message.content
    return msgs[:1] + [{"role": "system",
        "content": f"Summary of earlier conversation: {note}"}] + recent

for label, msgs in [("full history", history + [question]),
                    ("last 6 turns", keep_last(history, 6) + [question]),
                    ("summary + last 4", summarize_old(history, 4) + [question])]:
    answer, u = ask(msgs)
    usd = (u.prompt_tokens * PRICE_IN + u.completion_tokens * PRICE_OUT) / 1e6
    print(f"{label:<18} {u.prompt_tokens:>5} in  ${usd:.6f}  {answer[:70]}")

Results (Claude Sonnet 5, Oct 11, 2026)

Same model and same output length, with roughly a quarter of the bill. (Exact token counts vary a little between runs, because the summary text changes.)

The summary step costs extra: one call to GLM 5.3 Flash at about $0.00002. You only pay it when the history crosses your limit, not on every message.

The catch: quality

Cost is only half the test, so I added a memory check. At the very start of the chat, the user says: "My customer is Northwind, account ID 7731, net-45 terms." After 40 more turns, the bot is asked to repeat the ID and terms. Three runs each:

The plain sliding window is cheap, but it forgets anything older than the window, including the facts your user said only once. The summary keeps them at almost the same price. To make that reliable, tell the summarizer what to keep: "Keep names, IDs and terms."

A production version in 15 lines

MAX_TURNS = 8         # keep this many recent turns verbatim
SUMMARY_AT = 12       # compress once history passes this

def build_messages(system, turns, summary, new_msg):
    if len(turns) > SUMMARY_AT:
        summary = summarize(summary, turns[:-MAX_TURNS])  # cheap model
        turns = turns[-MAX_TURNS:]
    msgs = [{"role": "system", "content": system}]
    if summary:
        msgs.append({"role": "system", "content": f"Earlier: {summary}"})
    return msgs + turns + [new_msg], turns, summary

Store summary next to the conversation, so you only compress each part of the history once.

Other input-token leaks to check

• System prompts that grew to 2,000 tokens. They're sent on every call. Trim them, or use providers with prompt caching.
• RAG that stuffs 10 chunks when 3 would do. Rerank, then cut.
• Tool schemas. Every function definition is input on every call. Send only the tools that make sense for the current step.

I measured this through Qubax, an OpenAI-compatible API for 400+ models, often well below OpenRouter prices; the traffic numbers above come from its usage logs. The trimming code works with any OpenAI-compatible endpoint. Live model prices are on the price index.

FAQ

Is a sliding window good enough?

For short-lived chats, yes. But in our test it forgot an account ID given at the start 3 out of 3 times. If users state facts once and expect them to be remembered, summarize the older turns.

Which model should write the summary?

A cheap, fast one. The summary call in this test cost about $0.00002, and you only pay it when the history passes your limit.

Does prompt caching solve this?

Partly. Caching makes the repeated prefix cheaper, but long histories still add latency and can push you past context limits. Trimming and caching work well together.

References

• OpenAI, managing conversation state: https://platform.openai.com/docs/guides/conversation-state
• Anthropic prompt caching: https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching
• OpenRouter model pricing: https://openrouter.ai/models

How do you handle long chats: windows, summaries, or embeddings-based memory? Curious what holds up at scale.

Originally published at qubax.ai. Qubax gives you every top AI model with one API key, cheaper than OpenRouter.