context-triage 建議以逐回合分流取代 AI 智能體上下文摘要

A small decision model decides, turn by turn, what an LLM agent keeps, compresses or forgets.

Prompt caching is out of scope here. Editing or removing an earlier message changes the prompt from that point on, so content after the edit cannot reuse the cache. Unchanged prefixes, for example the system prompt and tools, can still be reused. Because triage removes the oldest turns, most of the conversation after the system prompt is affected. The exact impact depends on each provider's caching implementation, and a separate write-up will cover it.

The problem

Every turn and every tool call adds to an agent's context. That context is sent to the model on each call, so it drives the cost and latency of every call that follows. It also fills with material that no longer matters: old logs, abandoned directions, pleasantries. The noise sits next to the facts the agent needs.

The default answer is one big summary. When the context passes a limit, you ask an LLM to summarize everything so far and replace the history with the result. That is an all-or-nothing rewrite. It is lossy, it happens in a single irreversible step, and you cannot see why a given detail survived or vanished. It also spends your most expensive model on housekeeping.

One distinction up front. This is context management: what the model carries into its next call. It is not memory management, which is what an agent retains and retrieves across tasks or sessions. It complements memory systems such as Mem0 or Zep rather than replacing them, and could become a foundation for one.

The core idea: triage, not summarize

context-triage, built on 10xGraph, replaces the single rewrite with a separate decision for each unit of work. A unit is one turn: a user message and everything that answered it. (The Python package inside is still named unforget, its working name.) Each old turn gets one of three actions:

• keep: leave the turn as it is.
• compress: replace it with a 1 to 3 sentence note.
• drop: remove it. This is for greetings, thanks and small talk. Short messages are not automatically disposable: decisions, approvals, corrections and constraints ("Yes, use PostgreSQL", "Don't use AWS") must survive even when they are a few words.

Illustrative example (invented for explanation, not output from a run):

• Turn 1: "Hi, how are you?" -> drop
• Turn 2: "We are building a recruitment platform on Django and PostgreSQL." -> keep
• Turn 3: a 200-line API log -> compress to a one-sentence note, such as "Log showed repeated timeouts on one endpoint."
• Turn 4: "Thanks!" -> drop

After triage, the working context holds the Django and PostgreSQL turn verbatim, the one-sentence note, and the newest turns. The full conversation stays in persistent history.

A fourth action, archive, existed in the first version and is covered below as a planned return. The current code does not include it.

The design rests on a few principles.

Separate deciding from writing. A cheap decision model, Cloudflare Clef, decides. It does not write text. For each question it returns a probability for every option plus a confidence. A small writer model, claude-haiku-5-5, then writes notes, and only for the turns marked compress. The main model is never asked to summarize itself.

Log every decision with its confidence. Every decision can be recorded with Clef's suggestion and its model-reported confidence score. That gives you an audit trail of what was removed and how sure the model said it was. It also gives you a gate: if the top option falls below a threshold (0.7 here), the suggestion is ignored and the turn is kept. Be careful what the number means. A 0.95 does not mean the decision is right 95% of the time. The scores are not calibrated against a labelled set. The 0.7 gate is a fail-safe, not an accuracy guarantee.

Keep three views of one conversation.

• Persistent history: the thread's messages, stored by the checkpointer and never edited.
• Active model context: state.context, the only thing triage edits.
• User-facing history: in this setup the playground chat renders the stored thread messages, so the user still sees everything.

Take a medical assistant. A user asks for a poem and the bot refuses: "I only handle medical questions." Several turns later the context is over budget, and Clef may decide that exchange is chatter with nothing worth reusing. The model stops carrying it, and the user can still scroll up and read it. Note the difference between data retention and model accessibility. The data is retained, but the model cannot read removed turns back.

Work in whole turns. Each turn is an atomic unit. That keeps tool calls with their results and avoids leaving half-finished tool interactions in the context. Provider-specific ordering rules still have to be respected. The newest turns are never candidates, so the work in progress is safe.

Fail safe. If the decision model fails, the writer fails, or credentials are missing, the context is left unchanged and the reason is recorded. A failed trim costs you tokens, not correctness.

Measure, do not estimate. The trigger is the token usage the provider reported for the last reply, not a character count. The check runs only when that number exceeds the budget.

Write the policy in plain language. What counts as keep, compress or drop is not a trained classifier. It is the policy dict handed to the decision model:

ACTION_CRITERIA = {
    "keep": "Needed verbatim: the current work depends on its exact wording, code or numbers.",
    "compress": (
        "Useful but verbose, such as tool output, logs or listings, or it states preferences "
        "or decisions: a 1 to 3 sentence note would keep what matters."
    ),
    "drop": "Greetings, thanks, small talk or chatter with nothing worth reusing.",
}

Your team can read this, argue about it and change it without retraining anything. The policy routes preferences and decisions to compress, not drop. Whether the decision model honours that is untested.

Each turn is clipped to 800 characters (head and tail) before Clef sees it, and this is a named risk. A long requirement whose key constraint sits in the middle can be judged on its edges and dropped. That matters most for code, configs and technical specs. A possible fix is structure-aware extraction before classification, so the decision model sees the parts that carry the constraints.

Where it sits in the architecture

context-triage is a pluggable context manager in 10xGraph. The flow, in order:

1. The run ends and the answer has already been delivered to the user.
2. The manager reads the reported token usage of the latest reply. At or under budget, nothing happens.
3. Over budget, it groups the history into turns and offers the oldest candidates to Clef, all but the newest turns, in one call with one question per turn.
4. Any suggestion below the confidence threshold becomes keep.
5. The turns marked compress go to the writer in one batched call.
6. Compressed and dropped turns leave the working set, the notes are kept as a short block of context, and each decision is logged.

The answer has already streamed by the time this runs, so the user does not wait on it.

The cost shape is a plain sum:

total cost = classification + compression + cache impact

• Classification depends on the turns offered per check, the tokens per offered turn, and how often checks fire. Clef is priced at $0.24 per million input tokens, with no output charge.
• Compression is the writer's input and output tokens for the turns marked compress.
• Cache impact is the reuse lost after the context changes.

The net benefit depends on how many future model calls run on the smaller context. We make no savings claims.

Archive: planned, not shipped

The first version had a fourth action, archive, plus a recall tool. We removed both to keep the core loop easy to read. Bringing them back is the next step, because they answer the biggest weakness of the three-action version: once a turn is dropped or compressed, the model cannot get the detail back.

In that version an archived turn left the working set, the full turn was stored verbatim in thread state with a short label, and the context kept an index of labels. When the agent called recall with an item's number, the turn came back as a tool result and the archive itself was never modified.

Why this matters to a leader:

• It turns a one-way door into a two-way door. Where the policy is unsure whether a turn will matter again, it can archive instead of drop.
• It makes the policy safer to tune. A wrong call becomes one extra tool call instead of a lost fact, so tighter budgets carry less risk of silent loss.

A recall costs a tool call, with its latency and tokens, and the text re-enters the context when used. There is also a trust boundary. Recalled content is historical data, not fresh instructions. It can contain user instructions, tool output, external documents or prompt-injection attempts, so it must be returned as data, with that boundary made clear to the model.

What the one live run showed

We have one live run so far, a smoke test on 2026-11-01. It is the only evidence we have.

What it shows:

• The decision model classified simple turns.
• The gate held back a 0.546 decision.
• Drop shrank the working set to 2 messages.
• The full history stayed visible in the chat.

What it does not show:

• Correct decisions on complex conversations.
• Preserved facts.
• Compression quality.
• Better task completion.
• Real cost savings.

The compress path has not run live yet. We report no benchmarks or accuracy numbers.

The architectural lesson: provider-signed state

When we first removed turns from a conversation with extended thinking, the next request failed with a live 400: "Invalid signature in thinking block." Claude signs each thinking block against the whole prefix before it. Remove earlier turns and the blocks you kept no longer match the prefix they were signed against.

The lesson is general. Provider-signed state is bound to its history, so any system that edits context, whether by trimming, summarizing or triage, has to plan for it.

Our fix is blunt: after any change, strip every signed or redacted thinking block from the remaining messages, not just the first invalid one. The cost is that the model loses its earlier reasoning blocks. This strip route is what context-triage does through 10xGraph.

There is another route, which we verified in a separate small script on the direct Claude API with claude-sonnet-5-5 (not part of this project). In 10xGraph you pass these as extra Agent kwargs:

thinking={"type": "adaptive", "block_binding": {"prefix_mismatch_behavior": "drop_block"}},
extra_headers={"anthropic-beta": "thinking-binding-controls-2026-08-01"},

With drop_block, the request proceeds instead of erroring. It does not preserve the invalidated reasoning: the API discards the thinking blocks whose prefix no longer matches. Because triage removes the oldest turns, in practice most of the earlier reasoning is discarded. It is a beta, and we only verified it on the direct Claude API. The binding check is enforced by default for accounts created on or after 2026-08-31, and setting the field opts a request in.

How it compares

• One-shot summaries, such as the auto-compact in Claude Code, rewrite the whole history in one pass.
• Observation masking hides old tool output with a fixed rule. It is cheap but does not look at content.
• LLM-ranked condensers, such as OpenHands' attention condenser, keep the top N events and drop the rest, with no reasons.
• Memory stores, such as Mem0 and Zep, keep facts outside the live context.

context-triage is a context-management layer that can sit next to a memory store. It makes a separate decision per turn with a cheap model that is not your main model, needs no training, and logs its confidence scores. We do not claim to be first, and we have not measured it against any of these.

What a tech lead should weigh

Each of these is a decision point, not a footnote.

• Irreversibility for the model. In the current code, compressed notes lose detail and dropped turns are gone from the model's context for good. The thread history still holds everything, but the model cannot read it back. Until archive returns, decide whether your use case can live with this.
• Compression quality. The fail-safe covers a writer that fails, not one that succeeds with a wrong note: a changed number, a reversed decision, a missing identifier. A validation step for structured details (identifiers, dates, numbers, config values, explicit decisions) is an open item.
• Starvation in long threads. Each check offers the oldest candidate turns first, at most 64 per call. Kept turns are offered again on every over-budget reply. If the oldest 64 are all kept, newer turns beyond them are never evaluated. This is a known limitation of the current code. Possible fixes: a rotating window, recording when a turn was last evaluated, or re-evaluating a kept turn only when something changes.
• Trimming happens only at the end of a run. One long, tool-heavy run can grow past the budget until it finishes.
• A writer failure skips the whole trim, including decisions Clef already made. That is the safe choice; a flaky writer means no trim on those checks.
• A second vendor in the stack. Cloudflare is an extra account, an extra credential and an extra dependency. Compress still costs writer-model tokens.
• Thinking blocks are stripped after a trim unless you adopt the beta route above.
• Prompt caching. Editing old context may cancel any gain. We have not modelled it.

Questions to answer on your own workload

• How often does the decision model choose well, judged by someone who knows the task?
• What is lost in the compress notes, and does the agent ever need it back?
• What is the net cost once prompt caching is counted?
• How does task quality change, compared with doing nothing and with a plain summary?

The idea worth taking away

The architecture separates three responsibilities that a single summary call mixes together:

• Decision: what deserves space in the context.
• Transformation: compress what stays useful.
• Retention: keep the original independent of the model's active context.

This separation is the part worth taking away even if you never use Clef. You can swap the decision model, change the writer, or add an archive without touching the other two.

What we will measure next:

• The compress path in a live run, which has not happened yet.
• Compression validation for identifiers, dates, numbers, config values and explicit decisions.
• A fix for the 64-turn starvation problem.
• Clef's choice quality on real conversations.
• Real cost against a plain summary.
• Archive and recall: bringing them back, and measuring how often the agent actually recalls what was archived.
• The caching cost, in the follow-up write-up.

The code and setup instructions are on GitHub: github.com/Iamsdt/context-triage.

context-triage is built on 10xGraph, a framework for building production AI agents.