失敗程式碼留在對話紀錄為何會污染編碼智能體？錯誤追蹤剪除與累積歷史的比較

💡 AI Labo Benchmark Series — Experiment E19 (Ranked #4 out of 41 Patterns)

This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.

• 🚀 View All 41 Benchmark Results & Master Ranking
• 🛡️ Read How Our Proxmox & systemd Anti-Cheating Sandbox Works

Metadata & Experiment Identity

• Experiment ID: E19 (Backlog Candidate C015)
• Run ID: e19-20261028-vm223-01
• Sandbox Environment: Proxmox Isolated VM VM223 (ai-e19-223, 0 NICs, virtio-serial QGA relay, systemd UID separation)
• Model: gemini-3.8-flash (thinkingLevel: LOW)
• Pattern Under Test: Context Poisoning Prevention — "Error-Trace Pruning (Failed-Attempt Rollback) vs. Accumulated Failed History"

1. Executive Summary (TL;DR)

When a coding agent attempts a multi-file fix and the first patch fails tests, standard agent loops simply append the failed code diff (role: "model") and the verbose error stack trace (role: "user") to the conversation history (messages[]) and ask the LLM to try again.

This creates Self-Conditioning Bias (Context Poisoning): LLMs strongly anchor on their own prior generated code in the context window. Instead of rethinking the cross-module architecture, the agent keeps making narrow incremental tweaks on top of its broken first attempt while stack traces snowball across turns.

To test the antidote—Error-Trace Pruning (Failed-Attempt Rollback)—we benchmarked our 3-module payment settlement pipeline (validator.py, settlement.py, statement.py) inside isolated Proxmox VM VM223 across two modes:

• Mode A (AccumulatedFailedHistoryRunner): Retains failed partial patches and full error stack traces (36,606 chars on Turn 1; 36,264 chars on Turn 2) in the conversation history across retries.
• Mode B (PrunedFailedAttemptRunner): Immediately rolls back the workspace on test failure (git checkout), prunes the failed model output and raw stack trace from messages[], and appends only a 927-character (+392-token) structured negative lesson to the clean initial prompt.

Key Result

While both modes eventually passed all 20 / 20 (100%) hidden test cases, pruning failed turns in Mode B eliminated the second failed retry (3 calls ➔ 2 calls), compressed final-turn input tokens by -94.0% (38,134 ➔ 2,301 tokens), cut cumulative input tokens by -93.0% (59,944 ➔ 4,210 tokens, 14.24x efficiency), reduced API cost by -80.3% ($0.0558 ➔ $0.0110), and finished 1.57x faster.

2. Experiment Architecture & Rollback-Prune Workflow

+-------------------------------------------------------------------------+
| Proxmox Isolated VM223 (0 NICs, virtio-serial QGA, systemd UID split)   |
|  Target Task: 3-Module Settlement Pipeline Bugfix                       |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: AccumulatedFailedHistoryRunner (Keeps Failed Turns)       |  |
|  |  - Turn 1: Partial patch (validator.py) FAILS (+36,606 char trace)|  |
|  |  - Turn 2: Anchors on Turn 1; patches only settlement.py          |  |
|  |            FAILS again (+36,264 char trace; 19,912 in tokens)     |  |
|  |  - Turn 3: Submits all 3 files at 38,134 input tokens (59,944 sum)|  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: PrunedFailedAttemptRunner (Rollback + Prune Failed Turn)  |  |
|  |  - Turn 1: Partial patch (validator.py) FAILS (1,909 in tokens)   |  |
|  |  - Harness Action:                                                |  |
|  |      1. Roll back workspace files to clean snapshot               |  |
|  |      2. Delete Turn 1 model output & 36,606-char trace from chat  |  |
|  |      3. Append 927-char structured lesson (+392 tokens)           |  |
|  |  - Turn 2: Fresh retry at 2,301 input tokens --> Submits all 3    |  |
|  |            files in 1 shot (20 / 20 PASS, 4,210 cumulative in)    |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+

3. Turn-by-Turn Token Progression & Prompt Comparison

3.1 Turn-by-Turn Token & Cost Breakdown (usage.json)

3.2 What Mode A Kept in History (input-01.json / input-02.json)

In Mode A, after Turn 1 patched only validator.py, the harness appended 36,606 characters of unpruned diff frames and local variable dumps (MSG[2]), followed by another failed partial attempt (MSG[3]) and 36,264 characters of traces (MSG[4]):

=== FAILED ATTEMPT #1 FULL DIFF & TRACEBACK DUMP (UNPRUNED) ===
Status: FAILED (3 invariants broken)
[FAILED_PATCH_CONTEXT_FRAME_001] attempt=1 rejected_hunk_sha256=0000...0001 locals={'merchant_id': 'M-1', 'currency': 'USD', 'bps': 300, 'tier': 'ENTERPRISE'}
  File '/var/lib/ai-lab-e19-vm223-01/work_accumulated/settlement.py', line 26, in settle_batch
    AssertionError: Failed attempt #1 left residual bug in pipeline state.
... (FRAME_002 through FRAME_094: 36,000+ chars of repetitive frame dumps) ...

3.3 What Mode B Injected After Pruning (input-04.json)

In Mode B, as soon as Turn 1 failed, error_trace_pruner_v1 restored the workspace, deleted MSG[1] (the failed partial patch) and MSG[2] (the 36,606-char traceback) from messages[], and appended only this 927-character (+392-token) structured lesson to the original prompt:

{
  "concise_lessons_learned": [
    "Lesson 1 (validator.py): Partial patch failed because `settlement.py` and `statement.py` were not updated together; in `validator.py` use `ALLOWED_CURRENCIES=('USD','EUR','JPY')`, reject empty list & `bool`, allow `unit_cents>=0` & `0<=chargeback_bps<=1000`, and return a new 5-key dict.",
    "Lesson 2 (settlement.py): Preserve first-seen order (`for k in order:`), charge 10% reserve strictly when `chargeback_bps > 300`, apply `>= 15000` discount boundary (`GROWTH` 10%, `ENTERPRISE` 20% else 10%), and cap `rebate_applied = min(fee_rebate, after_discount)`.",
    "Lesson 3 (statement.py): Format item lines with `#` (`{merchant_id}#{currency}: ...`) and always prefix `-` on `TIER_DISCOUNT: -{tier_discount}` and `REBATE_APPLIED: -{rebate_applied}`."
  ],
  "pruning_engine": "error_trace_pruner_v1 (failed_code_rolled_back=true, stack_traces_pruned=true)",
  "remaining_issues": 4
}

With a clean 2,301-token context freed from its failed first attempt, Mode B immediately generated a complete, unified 3-module patch on Turn 2.

4. Practical Implementation Recipe: A 20-Line Rollback-and-Prune Loop

If you maintain a custom coding agent loop in Python, you can implement Error-Trace Pruning in 20 lines by combining git checkout with a lesson_log list instead of appending failed turns to messages:

import subprocess

def run_step_with_error_pruning(messages: list[dict], lesson_log: list[str], call_llm, run_tests) -> bool:
    # 1. Build prompt from clean base messages + 1-line negative lessons from prior failed attempts
    prompt_msgs = list(messages)
    if lesson_log:
        prompt_msgs[-1] = {
            **prompt_msgs[-1],
            "text": prompt_msgs[-1]["text"] + "\n\n[PRUNED_FAILED_ATTEMPT_LESSONS]:\n" + "\n".join(lesson_log),
        }
    model_reply = call_llm(prompt_msgs)

    # 2. Apply patch and run deterministic test suite
    passed, short_failure_reason, _raw_trace = run_tests(model_reply)
    if passed:
        messages.append({"role": "model", "text": model_reply})
        return True

    # 3. On failure: roll back workspace and DO NOT append model_reply or _raw_trace to messages!
    subprocess.run(["git", "checkout", "--", "."], check=True)
    lesson_log.append(f"- Attempt #{len(lesson_log) + 1} rolled back: {short_failure_reason}")
    return False

🔗 Continue Reading the Series

• 🚀 Master Benchmark Hub (All 41 Patterns Ranked): I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy
• 🛡️ Methodology & Sandbox Architecture: How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox
• 🥇 Rank #1 Deep Dive (E20): Stop Your Coding Agent From Stepping on the Same Rake Twice: Cold-Start Exploration vs. a 5-Item MEMORY.md
• 🥈 Rank #2 Deep Dive (E21): Why Feeding Full Dependency Files to Your Coding Agent Hurts Accuracy: Full Source vs. Tree-sitter Interface Skeletons
• 🥉 Rank #3 Deep Dive (E16): Stop Dumping Raw Pytest Logs Into Your Coding Agent: How a Structured JSON Test Filter Cuts Input Tokens by 94.1%