別讓編碼 Agent 重複踩坑：冷啟動探索與 5 項 MEMORY.md

💡 AI Labo Benchmark Series — Experiment E20 (Ranked #1 out of 41 Patterns)

This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.

• 🚀 View All 41 Benchmark Results & Master Ranking
• 🛡️ Read How Our Proxmox & systemd Anti-Cheating Sandbox Works

Metadata & Experiment Identity

• Experiment ID: E20 (Backlog Candidate C016)
• Run ID: e20-20261028-vm224-01
• Sandbox Environment: Proxmox Isolated VM VM224 (ai-e20-224, 0 NICs, virtio-serial QGA relay, systemd UID separation)
• Model: gemini-3.8-flash (thinkingLevel: LOW)
• Pattern Under Test: Letta / MemGPT & Claude Code CLAUDE.md / MEMORY.md Pattern — "Episodic Gotchas Memory vs. Cold-Start Exploration"

1. Executive Summary (TL;DR)

When running multiple tasks sequentially on the same repository, starting every agent session from scratch (Cold-Start Exploration) forces the model to rediscover non-obvious domain invariants and repository quirks ("gotchas") through repeated test failures—burning tens of thousands of tokens on stack traces and diluting model attention.

To test whether a compact Episodic Gotchas Memory (MEMORY.md) solves this, we benchmarked a 3-module payment settlement pipeline (validator.py, settlement.py, statement.py) containing 5 subtle repository gotchas (7 domain invariants) across two modes:

• Mode A (Cold-Start Exploration): No MEMORY.md injected; the agent explores the codebase and diagnoses failures via standard test/audit feedback.
• Mode B (Episodic Gotchas Memory): A 5-item MEMORY.md (+442 tokens) distilled from prior session failures is injected into the initial prompt.

Key Result

Injecting just +442 tokens of MEMORY.md reduced repository trap hits from 2 ➔ 0, achieving a 1-turn 20 / 20 (100%) pass on the hidden grading suite. Compared to Mode A (17 / 20 pass across 3 turns), Mode B cut cumulative input tokens by -95.5% (46,560 ➔ 2,078 tokens), API cost by -83.7% ($0.0454 ➔ $0.0074), and wall-clock time by -54.8% (2.21x faster).

2. Experiment Architecture & Repository Gotchas Under Test

+-------------------------------------------------------------------------+
| Proxmox Isolated VM224 (0 NICs, virtio-serial QGA, systemd UID split)   |
|  Target Task: 3-Module Settlement Pipeline Bugfix                       |
|               (validator.py / settlement.py / statement.py)             |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: ColdStartRunner (No MEMORY.md — Cold-Start Exploration)   |  |
|  |  - Turn 1: probe_repo_gotchas   (1,636 in / 28,317 chars trap log)|  |
|  |  - Turn 2: stage_partial_fix    (15,153 in / 27,975 chars log)    |  |
|  |  - Turn 3: submit_settlement    (29,771 in --> 17 / 20 passed)    |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: EpisodicGotchasMemoryRunner (5-Item MEMORY.md [+442 tok]) |  |
|  |  - Turn 1: Reads 5-item MEMORY.md & submits all 3 modules         |  |
|  |            (2,078 in / 1,560 out --> 20 / 20 passed on 1st try!)  |  |
|  +-------------------------------------------------------------------+  |
|                                    |                                    |
|                                    v                                    |
|  +-------------------------------------------------------------------+  |
|  | Independent Grader (labjudge20): 20 Hidden Unit & E2E Test Cases  |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+

The 5 Non-Obvious Repository Gotchas in the Initial Codebase

The initial workspace inside VM224 contains three payment modules with subtle domain bugs and Python traps that are easy to miss on a cold read:

1. validator.py (Transaction Validation & Normalization):
   • ALLOWED_CURRENCIES = ("USD", "EUR") is missing "JPY".
   • Empty list [] rejection is missing, and because bool is a subclass of int in Python (isinstance(True, int) == True), isinstance(v, bool) must be explicitly rejected before checking isinstance(v, int).
   • The bounds check if cnt <= 0 or cents <= 0 or not (0 <= bps < 1000): erroneously rejects unit_cents == 0 ($0 card authorization probes) and chargeback_bps == 1000 (inclusive upper bound).
   • It mutates caller dictionaries in-place (item["merchant_id"] = ...; out.append(item)) and leaks extra unvalidated keys (extra) into the output.
2. settlement.py (Batch Aggregation, Reserve & Tier Discounts):
   • Iterates with for k in sorted(order):, destroying first-seen (merchant_id, currency) ordering.
   • Applies the 10% chargeback reserve at chargeback_bps >= 300 instead of strictly > 300 (at exactly 300 bps, reserve must be 0).
   • Uses subtotal > 15000 instead of inclusive >= 15000, omits the 10% discount for ENTERPRISE below 15000, and forgets to cap rebate_applied = min(fee_rebate, after_discount).
3. statement.py (Canonical Statement Formatter):
   • Uses : instead of # between merchant ID and currency ({merchant_id}#{currency}:).
   • Formats zero discounts as "0" instead of preserving the mandatory leading minus sign (TIER_DISCOUNT: -0, REBATE_APPLIED: -0).

3. Inside the Prompts: What Mode B Saw vs. What Mode A Endured

3.1 The Exact 5-Item MEMORY.md Injected into Mode B (+442 tokens)

In Mode B (input-03.json), the harness prepended this exact 5-bullet # MEMORY.md section to the initial prompt:

# MEMORY.md — Repository Episodic Gotchas & Non-Obvious Invariants (Distilled from Session #1)
1. [GOTCHA-01 / validator.py] `ALLOWED_CURRENCIES` MUST include `"JPY"` (`("USD", "EUR", "JPY")`), empty `records` list MUST raise `ValueError`, and `isinstance(v, bool)` MUST be rejected before `isinstance(v, int)` because `bool` is a subclass of `int` in Python.
2. [GOTCHA-02 / validator.py] Zero `unit_cents == 0` and boundary `chargeback_bps == 1000` are VALID (`tx_count > 0`, `unit_cents >= 0`, `0 <= chargeback_bps <= 1000`). Always return a brand-new 5-key dict without mutating caller dicts.
3. [GOTCHA-03 / settlement.py] Preserve first-seen `(merchant_id, currency)` order (`for k in order:`, NEVER `sorted(order)`). The 10% reserve applies STRICTLY when `chargeback_bps > 300` (at `300` bps, `reserve_cents == 0`).
4. [GOTCHA-04 / settlement.py] Tier discount threshold `subtotal >= 15000` is INCLUSIVE: `GROWTH` gets 10% (`>= 15000`), `ENTERPRISE` gets 20% (`>= 15000`) or 10% (`< 15000`), and `rebate_applied = min(fee_rebate, after_discount)`.
5. [GOTCHA-05 / statement.py] Header delimiter between merchant and currency is `#` (`{merchant_id}#{currency}: tx=...`) and discount/rebate lines MUST always carry `-` even when zero (`TIER_DISCOUNT: -0`, `REBATE_APPLIED: -0`).

3.2 The 56,605-Character Failure Trace That Flooded Mode A (Cold-Start)

Without MEMORY.md, Mode A called probe_repo_gotchas on Turn 1 and stage_partial_fix on Turn 2, accumulating 28,317 characters on Turn 1 and 27,975 characters on Turn 2 (totaling 56,605 characters of verbose audit stack traces) inside its conversation history:

=== COLD-START REPOSITORY GOTCHA TRAP LOG (STAGE #1) ===
Status: FAILED (4 repository gotchas triggered due to missing cross-session MEMORY.md)
[COLD_START_GOTCHA_AUDIT_001] stage=1 invariant_probe=GOTCHA_02 trace_digest=0000000000000000000000000000000000000000000000000000000000000001
  File '/var/lib/ai-lab-e20-vm224-01/work_cold_start/settlement.py', line 21, in verify_repo_invariants
    AssertionError: Cold-start session rediscovered repository gotcha #2 via trial-and-error failure.
... (28,000+ chars of repetitive invariant audit traces) ...
TRIGGERED_GOTCHA [validator.py] (validator_contract_violation): Failed JPY currency, empty list check, bool rejection, zero unit_cents, 1000 bps, or mutated input dict.
TRIGGERED_GOTCHA [settlement.py] (settlement_order_or_reserve_boundary): Lines reordered by sorted(order) or 10% reserve charged at chargeback_bps == 300.
TRIGGERED_GOTCHA [settlement.py] (settlement_tier_discount_or_rebate_cap): Missed >=15000 inclusive discount boundary, ENTERPRISE 10% below 15000, or min(fee_rebate, after_discount) cap.
TRIGGERED_GOTCHA [statement.py] (statement_delimiter_or_signed_zero): Used ':' instead of '#' between merchant and currency or omitted '-' sign when discount/rebate is 0.

4. Code-Level Postmortem: Why Mode A Failed 3 out of 20 Test Cases

Comparing the final submitted code bundles (cold-start-bundle.json vs. episodic-memory-bundle.json) reveals the exact mechanism of Attention Dilution:

# ❌ Mode A (Cold-Start, Turn 3 Final Submission — from cold-start-bundle.json)
        if cnt <= 0 or cents <= 0 or not (0 <= bps <= 1000):
            raise ValueError("numeric bounds violated")
        out.append({
            **item,
            "merchant_id": mid.strip().upper(),
            "currency": cur.strip().upper(),
        })

# ✅ Mode B (Episodic Gotchas Memory, Turn 1 Final Submission — from episodic-memory-bundle.json)
        if cnt <= 0 or cents < 0 or not (0 <= bps <= 1000):
            raise ValueError("numeric bounds violated")

        out.append({
            "merchant_id": mid.strip().upper(),
            "currency": cur.strip().upper(),
            "tx_count": cnt,
            "unit_cents": cents,
            "chargeback_bps": bps,
        })

By Turn 3, Mode A was carrying 29,771 input tokens of noisy logs. While it managed to fix settlement.py, statement.py, "JPY", bool rejection, and chargeback_bps <= 1000, its diluted attention missed two fine-grained details in validator.py:

1. Left cents <= 0 instead of cents < 0: Rejecting unit_cents == 0 ($0 auth probes) caused both unit test validator_05_zero_unit_cents_allowed_negative_rejected and E2E test e2e_03_zero_amount_auth_statement to fail.
2. Used `{item, ...} spread instead of constructing a clean 5-key dict:** Extra unvalidated keys from the input dict leaked into the normalized output, failing **validator_01_normalize_and_non_mutation`**.

Mode B, reading [GOTCHA-02 / validator.py] inside a clean 2,078-token Turn 1 prompt, nailed every single invariant on the first try.

All 20 Hidden Test Case Results (Mode A vs. Mode B)

5. Practical Takeaway: How to Structure Your Own MEMORY.md

A 442-token investment saved 44,482 input tokens (a 100x+ token ROI) while eliminating 3 subtle edge-case bugs.

To get the same result in your own repositories with Claude Code, Cursor, or custom agents:

• Don't turn CLAUDE.md or MEMORY.md into a 500-line textbook. (As we proved in E35, monolithic instruction files hurt performance.)
• Instead, maintain a 10-to-20 line MEMORY.md containing only [GOTCHA-ID / filename] 1-line invariant entries distilled from actual test failures or PR review catches:

# MEMORY.md (Repository Gotchas — Keep under 20 lines)
- [GOTCHA-01 / validator.py] `isinstance(x, bool)` is True for `int` in Python; always reject `bool` before checking `isinstance(x, int)`.
- [GOTCHA-02 / validator.py] `unit_cents == 0` ($0 auth) and `chargeback_bps == 1000` are valid; construct a new 5-key dict (never `**item` or mutate in-place).
- [GOTCHA-03 / settlement.py] Keep first-seen order (`for k in order:`); apply 10% reserve strictly when `chargeback_bps > 300`.
- [GOTCHA-04 / statement.py] Use `#` between merchant and currency (`{mid}#{cur}:`), and format zero discount/rebate with a leading minus (`-0`).

🔗 Continue Reading the Series

• 🚀 Master Benchmark Hub (All 41 Patterns Ranked): I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy
• 🛡️ Methodology & Sandbox Architecture: How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox