Claude 上下文窗口實戰：快取、分塊與成本

A bigger context window raises a design question that didn't exist a couple of years ago: if the model can read 100 pages at once, do you still need retrieval?

Sometimes you don't. Often you do. This post walks through three ways to give Claude your documents (stuff, cache, retrieve), with minimal Python for each and a rough way to compare cost. The code is a sketch written from the Anthropic documentation, not a tested library, so check parameter names and pricing against the current docs before you copy anything.

Start by counting tokens

Before choosing an approach, find out how big your material really is. The SDK can count tokens for a request without running it:

import anthropic

client = anthropic.Anthropic()

count = client.messages.count_tokens(
    model="claude-sonnet-5-5",
    system="You answer questions about our policy handbook.",
    messages=[{"role": "user", "content": handbook_text + "\n\nQuestion: ..."}],
)
print(count.input_tokens)

If the number is a small fraction of the model's context window, long context is on the table. If it's close to the limit, or your corpus is many times larger, retrieval is the only real option.

Option 1: put it all in the prompt, and cache it

For a stable document set that many requests share, such as a handbook, a product spec, or a contract template, you can send the whole thing each time and mark it for caching. Repeated requests then reuse the processed prefix instead of paying full price for it again.

response = client.messages.create(
    model="claude-sonnet-5-5",
    max_tokens=500,
    system=[
        {"type": "text", "text": "You answer questions about our policy handbook."},
        {
            "type": "text",
            "text": handbook_text,
            "cache_control": {"type": "ephemeral"},
        },
    ],
    messages=[{"role": "user", "content": question}],
)

usage = response.usage
print(usage.cache_creation_input_tokens, usage.cache_read_input_tokens)

Three habits make caching work:

• Put stable content first and changing content last. The cache matches on a prefix. A timestamp or a user name near the top defeats it.
• Watch the usage fields. cache_read_input_tokens should be large on the second and later calls. If it stays at zero, something in your prefix is changing.
• Know the limits. There's a minimum prefix length before caching applies, and cached entries expire after a short time unless they're used again. Both vary by model, so check the docs.

Option 2: chunk and retrieve

When the corpus is large, changes often, or has to respect per-user permissions, retrieval is the better fit. You split documents into chunks, find the few that matter for each question, and send only those.

A simple chunker that keeps paragraphs together and adds a small overlap:

def chunk(text, max_chars=3000, overlap=300):
    paragraphs = text.split("\n\n")
    chunks, current = [], ""
    for p in paragraphs:
        if len(current) + len(p) > max_chars and current:
            chunks.append(current)
            current = current[-overlap:]
        current += p + "\n\n"
    if current.strip():
        chunks.append(current)
    return chunks

Then retrieval and prompt assembly. embed here stands for whatever embedding model you use, since that's a separate choice:

import numpy as np

def top_chunks(question, chunks, chunk_vectors, k=5):
    q = np.array(embed(question))
    scores = [float(np.dot(q, v) / (np.linalg.norm(q) * np.linalg.norm(v)))
              for v in chunk_vectors]
    best = sorted(range(len(chunks)), key=lambda i: scores[i], reverse=True)[:k]
    return [chunks[i] for i in best]

def answer(question, chunks, chunk_vectors):
    context = "\n\n---\n\n".join(top_chunks(question, chunks, chunk_vectors))
    return client.messages.create(
        model="claude-sonnet-5-5",
        max_tokens=500,
        system="Answer only from the provided excerpts. If they don't contain the answer, say so.",
        messages=[{"role": "user", "content": f"Excerpts:\n{context}\n\nQuestion: {question}"}],
    )

Retrieval has its own failure modes: the right chunk may not rank in the top five, and answers that span several sections can get cut. Budget time to evaluate it, not just build it.

A rough cost comparison

Prices change, so I won't print any. This helper takes your own numbers. The write and read multipliers describe how cached input is billed relative to normal input, so take them from the current pricing page:

def monthly_input_cost(
    doc_tokens, calls_per_month, price_per_mtok,
    cache_write_mult=None, cache_read_mult=None,
):
    full = doc_tokens * calls_per_month / 1_000_000 * price_per_mtok
    if cache_write_mult is None:
        return full  # no caching
    # simplified: one cache write, the rest reads (ignores expiry gaps)
    write = doc_tokens / 1_000_000 * price_per_mtok * cache_write_mult
    reads = doc_tokens * (calls_per_month - 1) / 1_000_000 * price_per_mtok * cache_read_mult
    return write + reads

Run it for your own document size and question volume, then compare against the retrieval path, where each call sends only a few thousand tokens. Two cautions. Caching entries expire, so a low-traffic app will rewrite the cache more often than this simplified function assumes. And retrieval has costs beyond tokens: embeddings, a vector store, and the engineering time to keep the index fresh.

How I choose

Whichever you pick, test with your own questions. Models can miss details buried deep inside very long inputs, and a small test set of real questions with known answers will tell you more than any benchmark.

What to log

Whatever the approach, record these per call: input tokens, cached tokens read and written, output tokens, latency, and which chunks or documents were included. When cost or quality moves, those fields tell you why.

I've gone deeper on window sizes and cost patterns in Claude context window limits and cost-smart patterns.

How big is the biggest document set you've tried to put in a single prompt, and did it hold up? I'd like to hear what worked.

I'm Darius Tran, an AI Implementation Specialist at AI Hive, working on getting AI agents into production for enterprise teams.