Context Rot：如何測試應用程式的長上下文退化並精簡模型讀取內容

Your help-desk bot resends the whole conversation every turn. In message three the customer said she is on the annual plan. By message forty she asks for a refund, and the bot quotes the monthly policy. The fact was in the prompt the whole time; the model just did not use it.

By the end of this post you will have two small scripts that measure whether long context hurts your app, and a short list of moves that trim what the model reads.

Here is the video version, with the drawings, if you would rather watch first:

What is actually in the context

The context is the set of tokens included when you sample from a model: system prompt, chat so far, tool results, documents. And the model keeps no memory of the last turn. With the Messages API you send the full conversation history on every call.

So turn forty carries all thirty-nine messages before it, and the number of input tokens per call only goes up. Meanwhile windows got huge: it is common for modern models to accept millions of tokens. The easy move is to paste everything, because it fits.

Fitting in the window and being read well are different things. As the number of tokens in the context grows, the model's ability to recall information from it decreases. That is context rot. It emerges across all models, though some degrade more gently than others, so switching models will not make it go away.

Step 1: log the size of every call

Start with the cheapest check. Anthropic's token counting endpoint is free to use, subject to rate limits. Count what your bot is about to send, and log it against the turn number (TypeScript, run as an ES module so top-level await works):

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic(); // reads ANTHROPIC_API_KEY from the environment
const MODEL = process.env.MODEL!; // set this to the model id you use

type Msg = { role: "user" | "assistant"; content: string };

async function count(messages: Msg[]): Promise<number> {
  const res = await client.messages.countTokens({ model: MODEL, messages });
  return res.input_tokens;
}

const history: Msg[] = [];
for (let turn = 1; turn <= 40; turn++) {
  history.push({ role: "user", content: `message ${turn}: some customer text` });
  console.log(turn, await count(history)); // what the bot is about to send
  history.push({ role: "assistant", content: `reply ${turn}: some bot text` });
}

Plot that against the turn number. In a chat that resends everything it never goes down. That is the pile the model has to read through each time.

Why the pile hurts

Anthropic's picture: in a transformer every token can attend to every other token, which gives n squared pairwise relationships for n tokens. Double the tokens and the pairs grow about four times. The fine print: that is the cost of full attention, and some models use cheaper attention variants, so treat it as a way to picture the problem, not a law for every model.

As context grows, the model's ability to capture those relationships gets stretched thin. Models also develop their attention patterns from training data where shorter sequences are more common than longer ones. The result is an attention budget: every new token you add depletes it by some amount. I like that framing because it turns every pasted line into a cost.

What the research shows

Chroma's context rot report, from July 2025, evaluated 18 models. They held task complexity constant and varied only the input length. Performance degraded as input grew, even on simple tasks, and in non-uniform ways.

One of their tests looks a lot like our support bot: long chat histories and one question about them. A naive chat memory puts the full chat history into the prompt. The focused prompts, with only the relevant parts, averaged about 300 tokens. The full prompts averaged about 113k tokens. Across all models, performance was significantly higher on the focused prompts. One explanation they give: irrelevant context adds a step, finding what is relevant, so the model does two tasks at once.

A second study, NoLiMa, tested 13 models that claim at least 128K tokens of context. At 32K tokens, 11 of them dropped below 50% of their short-length baselines.

My takeaway: do not trust the window size. Test the length you actually send.

Step 2: run a needle test

This is the check I would start with: an afternoon of work, and it ends the arguments. Hide one fact your app must find, at a few lengths and positions, ask the same question each time, and score the answers. A grid of three lengths by three positions is nine prompts.

Two details matter. First, the classic needle-in-a-haystack test is one that models score near-perfectly on, because it mostly tests direct matching of words. Second, when the needle and the question are less similar, performance degrades faster as length grows. So word the fact differently from the question: "billed once a year" versus "what plan is she on?"

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();
const MODEL = process.env.MODEL!;

const NEEDLE = "By the way, I get billed once a year for this.";
const QUESTION = "Is this customer on a monthly or an annual plan? Answer with one word.";
const FILLER = [
  "Thanks, that worked. Can you also check the shipping address?",
  "The invoice PDF looks wrong on my phone.",
  "I reset my password twice and still see the old dashboard.",
  "Is there a way to export my tickets as a spreadsheet?",
];

async function count(text: string): Promise<number> {
  const res = await client.messages.countTokens({
    model: MODEL,
    messages: [{ role: "user", content: text }],
  });
  return res.input_tokens;
}

async function build(targetTokens: number, position: number): Promise<string> {
  const lines: string[] = [];
  while ((await count(lines.join("\n") || "x")) < targetTokens) {
    for (let i = 0; i < 100; i++) lines.push(FILLER[lines.length % FILLER.length]);
  }
  lines.splice(Math.floor(lines.length * position), 0, NEEDLE);
  return lines.join("\n");
}

async function ask(chat: string): Promise<string> {
  const res = await client.messages.create({
    model: MODEL,
    max_tokens: 2000,
    messages: [{ role: "user", content: `${chat}\n\n${QUESTION}` }],
  });
  return res.content
    .map((b) => (b.type === "text" ? b.text : ""))
    .join("")
    .trim()
    .toLowerCase();
}

console.log("tokens  position  correct");
for (const target of [2_000, 20_000, 100_000]) {
  for (const position of [0, 0.5, 1]) {
    const answer = await ask(await build(target, position));
    console.log(`${String(target).padStart(7)}  ${String(position).padStart(8)}  ${answer.includes("annual")}`);
  }
}

Mind the cost: the full grid sends a few hundred thousand input tokens, so watch your bill and rate limits. That prints one small table. Rows that flip from true to false as the length grows are your context rot. Repeat the question a few times per cell if your model is not deterministic, and swap the repeated filler for real chat snippets from your own logs when you can.

Position matters in some setups and not in others. The 2023 Lost in the Middle paper moved the answer around inside long multi-document prompts. Performance was often highest when the relevant information was at the beginning or end, and degraded significantly when it was in the middle. That was on 2023 models, so newer ones vary: when Chroma hid one simple fact in filler, they found no notable variation by needle position. That is why the grid has three positions. Your model decides.

Step 3: run focused versus full on your eval set

Chroma's idea, on your data. For each question in your eval set, run it twice: once with only the relevant messages, once with everything. If the focused version clearly wins, you have found your rot.

type Msg = { role: "user" | "assistant"; content: string };
type EvalQuestion = { question: string; relevantMessages: Msg[]; allMessages: Msg[] };

async function runEval(
  questions: EvalQuestion[],
  askFn: (messages: Msg[], question: string) => Promise<string>,
  scoreFn: (q: EvalQuestion, answer: string) => number,
) {
  let focusedWins = 0;
  for (const q of questions) {
    const focused = scoreFn(q, await askFn(q.relevantMessages, q.question));
    const full = scoreFn(q, await askFn(q.allMessages, q.question));
    if (focused > full) focusedWins++;
  }
  return { focusedWins, total: questions.length };
}

Plug in your own askFn (it takes a message list and a question) and scoreFn. A clear gap in favour of the focused runs is the signal to trim.

Distractors: why stale facts can become a trap

Back to the bot quoting the monthly policy. That is a distractor: text that is topically related to the answer but does not quite answer the question. Chroma found that even a single distractor reduces performance relative to the needle alone, and four distractors compound it. The impact of distractors amplifies as input length grows.

Models fail in two styles here. Some abstain and say no answer can be found. Others answer confidently and wrongly. Our bot did the second one. Old plans and cancelled orders can act as distractors in a long chat, so I would remove stale facts, not just add new ones.

Trim what the model reads

The goal, in Anthropic's words, is the smallest possible set of high-signal tokens that maximizes the chance of the outcome you want. Five moves:

1. Retrieve, do not paste. Keep lightweight identifiers such as file paths, stored queries or links, and let the agent load data at runtime with tools. For the bot, that is a search tool over past tickets instead of every ticket pasted in.
2. Clear old tool results. Tool result clearing is one of the safest, lightest forms of compaction. A search result from turn three is noise by turn thirty.
3. Keep a notes block. In structured note-taking the agent writes notes outside the context window and pulls them back in later. Here is the shape:

type Msg = { role: "user" | "assistant"; content: string };

const notes: Record<string, string> = { plan: "annual", lastOrder: "blue headphones" };

function buildMessages(notes: Record<string, string>, recentTurns: Msg[], question: string): Msg[] {
  const pinned = "Key facts:\n" + Object.entries(notes).map(([k, v]) => `- ${k}: ${v}`).join("\n");
  return [...recentTurns, { role: "user", content: `${pinned}\n\n${question}` }];
}

notes.plan = "monthly"; // overwrite on change: the old value is gone

1. Compact near the limit. Compaction means summarizing a conversation that is nearing the window limit and starting a new window with the summary. The catch is that a summary can drop the one detail you needed, so I would keep the notes block too.
2. Use a sub-agent for the big digging. Sub-agents handle focused tasks with clean context windows. One may use tens of thousands of tokens and return a condensed summary, often 1,000 to 2,000 tokens.

If I had to pick one, it is the notes block, because it fixes the stale-fact trap.

Layout helps as well. For inputs of 20k+ tokens, Anthropic's prompting guide says to put long documents near the top of the prompt, above your query, instructions and examples. Putting the query at the end can improve response quality by up to 30% in their tests. It also suggests asking the model to quote the relevant parts first, which helps it focus. For the bot, I would put the key customer facts in a short block right above the question and ask for the quote.

A bigger window will not save you: Anthropic expects context windows of all sizes to remain subject to context pollution.

What to do on Monday

1. Log input tokens per call and plot them against the turn number.
2. Run the needle test with a fact worded differently from the question.
3. Run your eval set focused versus full.
4. Add a notes block and update it when a fact changes.
5. Put the long documents first and the question last.

The drawn version of all this, with the support bot and the attention budget, is in the video above.

The short version, the code and a quick quiz are on doodle.monster: https://doodle.monster/videos/context-rot-why-longer-prompts-make-your-agent-w