Context Rot：一個尚未解決的 AI 問題？

Large Language Models (LLMs) are very good at understanding the intent of the user and can give exceptionally good and accurate responses at the start of a session. When you are starting out a fresh session, it does not matter whether you are talking about science, literature, mathematics, or planning to find extraterrestrial life in our galaxy — it will give you the response you are looking for.

But as the session moves forward and the input window grows, the responses seem irrelevant or vague, and the results no longer align with your mental image.

Most people will think there is some issue in their prompt or writing style, or that maybe they are not clarifying the idea properly, and they end up confusing the LLM even more as the session decays.

But the issue is not in your idea or writing style. It is in the LLM itself. The information you are giving is not wrong, but the LLM starts to pay less attention with each passing input as the context window grows. This issue is known as Context Rot.

What is Context Rot?

Context Rot is a flaw in the architectural design of LLMs, and it has been there for the last 5–6 years. It is a phenomenon that degrades the performance of LLMs whenever they try to process a long context.

The problem is that the information is present in the context, but the model is unable to identify it within the context window.

Context Window is the working memory of an artificial intelligence or LLM — it is the maximum amount of data that an AI can see or remember during an active session.

The explanation for this problem is quite simple. The architecture of AI is designed in such a way that it naturally focuses on the beginning and end of the input and neglects the middle part. This means adding more context or information actually reduces the accuracy of the system — the attention of the AI becomes vague, and it starts hallucinating. This is known as "Lost in the Middle."

Researchers discovered Lost in the Middle using a famous technique called the "Needle in the Haystack" (NIAH) test, which was introduced by AI educator and YouTuber Greg Kamradt.

Needle in the Haystack

It is a test conducted on LLMs to evaluate the ability of a model to retrieve a specific, isolated piece of information from a large context.

As AI companies expanded and started marketing their AI products with large context windows — such as 1M, 2M, and 10M tokens — researchers needed a benchmark test to validate the claims behind these windows. During the research, they started noticing the Context Rot flaw, and to prove it, they introduced this technique to show how an AI becomes irrelevant even when its context window spans millions of tokens.

How the test works

• The Haystack: A massive amount of content is loaded into the LLM's context window, such as an essay collection, documentation, or a research paper.
• Needle: A random, out-of-context question is given to the LLM to describe or solve within the given content. This tests not only the model's matching ability but also its thinking capability.
• Depth: The needle is inserted into the haystack at different depths, so the result is an average across many positions — some easy, some difficult.
• Length: The haystack amount starts from 2,000 tokens and goes up to the outer bounds of the model's context window.
• Results: The outcome of NIAH is obtained after a recursive process of testing and evaluating different combinations of depth and context length.

Lost in the Middle

Even though NIAH is now used as an industry-standard tool, and top AI companies use it to showcase their powerful models, they are still unable to solve the biggest problem in AI: "Lost in the Middle." Models are still not able to manage such large chunks of data.

To understand how the Lost in the Middle problem arises, we need to understand the under-the-hood functioning of an LLM. (To learn about transformer architecture in detail, visit Great Learning.)

Large Language Models use the transformer architecture, which processes information and calculates relationships across an entire block simultaneously — unlike a human, who reads a sentence left to right. The model focuses on and emphasizes the beginning and end of the input, as these parts tend to reveal the user's intent or purpose.

This architecture is the reason models do exceptionally well on early, standalone NIAH tests but fail completely during real-world, complex reasoning conversations, where key details get buried among the instructions.

As there is no solution to this problem yet, you can use these workarounds in the meantime to increase your productivity with AI.

Techniques to Work Around Context Rot

1. Start Fresh Session

Instead of mentioning every aspect of your goal in one go, divide your idea or work into small, manageable tasks and use a new session each time to complete a single task. This way, the LLM will only try to complete your intent in the most accurate manner and will not get confused by multiple requests stacked on top of each other.

2. Handoff Approach

This is a context management technique where you summarize the LLM's output midway through the session and pass that summary down to a new session. It keeps the LLM's decision-making fresh.

3. RAG (Retrieval-Augmented Generation)

This is the most common technique used in enterprise settings to handle data at scale. In this technique, data is fetched from an external source or database in chunks using vectorization, and only the relevant chunk is fed into the LLM for that specific input. (If you need an in-depth explanation of RAG, visit GeeksForGeeks.)

4. RLM (Recursive Language Models)

This is one of the most recent approaches to context rot, described in the paper "Recursive Language Models" by Alex L. Zhang, Tim Kraska, and Omar Khattab at MIT.

In this work, the researchers explain that context rot can be reduced by storing data as an out-of-context variable inside an isolated Python REPL (Read-Eval-Print Loop) environment. The root LLM never sees the full raw text and only receives small, constant-sized metadata about it; Python execution commands are then used to read, slice, and filter keywords to actively map out the massive variable. The printed output of its own code feeds back into its next logical step.

If the model finds a dense, 10,000-word block of code or text that requires isolated semantic focus, it writes a line of code to call a fresh sub-instance of the LLM. This child model is born with a blank context window, and the root model aggregates the returned values from these sub-instances into the final response.

FAQs

Q: Is Context Rot the same thing as hallucination?

Not exactly. Hallucination is when the model makes up information that was never there in the first place. Context Rot is different — the correct information is present in the context window, the model just fails to pay attention to it because it is buried in the middle of a long input. So Context Rot can end up causing hallucination, but the root cause is attention, not invention.

Q: If I use a model with a bigger context window, will Context Rot go away?

No, and this is the part most people get wrong. A bigger context window only increases how much data the model can technically hold, it does not fix how the model pays attention to that data.

Q: How do I know if I am hitting Context Rot in my own conversations?

The clearest sign is when the model starts giving vague, generic, or slightly-off responses after a long session, even though your prompt is clear and you have already given it all the information it needs.

Q: Is there any permanent fix to Context Rot coming?

Right now, no. It is still an open architectural limitation, which is why researchers are actively working on new approaches.

References

• Hong, K., Troynikov, A., and Huber, J. (2025, July 14) — Context Rot: How Increasing Input Tokens Impacts LLM Performance.
• Trollope, R. (2026, May 18) — Context Is All You Need: Introducing Redis Iris.
• Zhang, A. L., Kraska, T., and Khattab, O. — Recursive Language Models.

Thanks for reading.