為何向 AI 編碼智能體提供完整依賴檔案會降低準確度：完整原始碼與 Tree-sitter 介面骨架比較

💡 AI Labo Benchmark Series — Experiment E21 (Ranked #2 out of 41 Patterns)

This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.

• 🚀 View All 41 Benchmark Results & Master Ranking
• 🛡️ Read How Our Proxmox & systemd Anti-Cheating Sandbox Works

Metadata & Experiment Identity

• Experiment ID: E21 (Backlog Candidate C017)
• Run ID: e21-20261028-vm225-01
• Sandbox Environment: Proxmox Isolated VM VM225 (ai-e21-225, 0 NICs, virtio-serial QGA relay, systemd UID separation)
• Model: gemini-3.8-flash (thinkingLevel: LOW, temperature: 0.0)
• Pattern Under Test: Aider Repo-Map & Tree-sitter Skeleton Compression — "Full Dependency Implementation Context vs. Interface-Only Skeleton Context"

1. Executive Summary (TL;DR)

When an AI coding agent edits a module that depends on several peripheral files, a common approach is to dump the entire source code of all imported dependencies into the prompt so the model "has full context."

In practice, feeding full implementation bodies of peripheral dependencies is actively counterproductive. Legacy helper logic, internal comments, and verbose function bodies bloat the context window and trigger Attention Dilution, causing the model to miss strict boundary contracts in the files it actually needs to edit.

To quantify this, we benchmarked a 3-module payment settlement pipeline (validator.py, settlement.py, statement.py) inside isolated Proxmox VM VM225 across two harness modes:

• Mode A (FullDependencyContextRunner): Loads the full implementation bodies of peripheral dependency modules and raw diagnostics (78,719 chars of tool output across 3 turns).
• Mode B (TreeSitterSkeletonContextRunner): Elides function implementation bodies (...) using an AST / Tree-sitter projection, injecting only public function signatures, type annotations, and contract docstrings (+673 tokens on Turn 1).

Key Result

Passing only the Tree-sitter Interface Skeleton (2,316 input tokens) enabled a 1-turn 20 / 20 (100%) pass on the hidden test suite. In contrast, Mode A's full dependency dump consumed 45,270 input tokens across 3 turns and stalled at 17 / 20 (85.0%) due to attention dilution. Mode B cut cumulative input tokens by -94.9% (19.55x reduction), API cost by -82.6% ($0.0446 ➔ $0.0078), and execution time by -57.2% (2.34x faster).

2. Experiment Architecture & Harness Workflow

+-------------------------------------------------------------------------+
| Proxmox Isolated VM225 (0 NICs, virtio-serial QGA, systemd UID split)   |
|  Target Task: 3-Module Settlement Pipeline Refactoring                  |
|               (validator.py / settlement.py / statement.py)             |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: FullDependencyContextRunner (Unfiltered Full Source Dump) |  |
|  |  - Turn 1: load_full_dependency_dump (1,643 in / 39,373 chars)    |  |
|  |  - Turn 2: stage_dependency_patch    (14,712 in / 39,027 chars)   |  |
|  |  - Turn 3: submit_skeleton_guided_bundle (28,915 in --> 17 / 20)  |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: TreeSitterSkeletonContextRunner (Signature & Contract)    |  |
|  |  - Turn 1: Reads AST Interface Skeleton (+673 tok) & submits all  |  |
|  |            3 modules (2,316 in / 1,611 out --> 20 / 20 on Turn 1) |  |
|  +-------------------------------------------------------------------+  |
|                                    |                                    |
|                                    v                                    |
|  +-------------------------------------------------------------------+  |
|  | Independent Grader (labjudge21): 20 Hidden Unit & E2E Test Cases  |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+

3. Inside the Prompts: Tree-sitter Skeleton vs. Full Implementation Dump

3.1 The Exact Tree-sitter Interface Skeleton Injected into Mode B (input-03.json)

Instead of dumping internal helper functions, Mode B's harness projected the target modules and their contracts into a compact Interface Skeleton (2,316 total input tokens on Turn 1) with implementation bodies folded into ...:

# REPO-MAP INTERFACE SKELETON (AST / Tree-sitter Signature & Contract Projection — Implementation Bodies Elided)

# module: validator.py (PUBLIC INTERFACE SKELETON)
ALLOWED_CURRENCIES: tuple[str, ...] = ("USD", "EUR", "JPY")
def validate_txns(raw_txns: list[dict]) -> list[dict]:
    """Contract:
    1. Reject empty list or non-list with ValueError.
    2. Strip & uppercase `merchant_id` (non-empty str) and `currency` (must be in ALLOWED_CURRENCIES: USD, EUR, JPY).
    3. Reject `isinstance(v, bool)` before `isinstance(v, int)` on `tx_count`, `unit_cents`, `chargeback_bps`.
    4. Bounds: `tx_count > 0`, `unit_cents >= 0` (zero allowed!), `0 <= chargeback_bps <= 1000` (1000 inclusive!).
    5. Return a brand-new 5-key dict list without mutating caller dicts.
    """
    ...

# module: settlement.py (PUBLIC INTERFACE SKELETON)
ALLOWED_TIERS: tuple[str, ...] = ("STANDARD", "GROWTH", "ENTERPRISE")
def settle_batch(txns: list[dict], tier: str = "STANDARD", fee_rebate: int = 0) -> dict:
    """Contract:
    1. Preserve first-seen `(merchant_id, currency)` order (`for k in order:`, NEVER `sorted(order)`).
    2. Chargeback reserve: 10% (`(gross * 10) // 100`) STRICTLY when `chargeback_bps > 300` (at 300 bps, reserve == 0).
    3. Tier discount threshold `subtotal >= 15000` is INCLUSIVE:
       - `GROWTH`: 10% if `subtotal >= 15000` else 0.
       - `ENTERPRISE`: 20% if `subtotal >= 15000` else 10%.
    4. Rebate cap: `rebate_applied = min(fee_rebate, after_discount)`.
    """
    ...

# module: statement.py (PUBLIC INTERFACE SKELETON)
def render_statement(batch: dict) -> str:
    """Contract:
    1. Line format: `{merchant_id}#{currency}: tx={tx_count} unit={unit_cents} bps={chargeback_bps} = {line_total}` (delimiter is `#`, NOT `:`).
    2. Discount & rebate lines MUST always carry a leading `-` sign even when 0 (`TIER_DISCOUNT: -0`, `REBATE_APPLIED: -0`).
    """
    ...

3.2 What Mode A Received: 78,719 Characters of Unfiltered Dependency Implementation Noise

In Mode A (FullDependencyContextRunner), calling load_full_dependency_dump on Turn 1 and stage_dependency_patch on Turn 2 flooded the context with 39,373 chars and 39,027 chars of internal helper bodies, bitwise checksum loops, and deprecated v1 comments:

=== FULL DEPENDENCY IMPLEMENTATION DUMP & INTERFACE MISMATCH LOG (STAGE #1) ===
Status: MODE_A_TRAP_HIT (4 contract invariants unresolved in FullDependencyContextRunner)
# RAW DEPENDENCY IMPLEMENTATION DUMP (UNFILTERED FULL SOURCE OF 12 INTERNAL HELPER MODULES)
# WARNING: Contains legacy internal helpers, deprecated rounding tables, and verbose implementation bodies.

# --- internal_dep_module_01.py (Full Implementation Body) ---
def _internal_helper_pipeline_stage_01(payload_buffer, flags=17):
    """Internal helper implementation body #01 (non-public implementation details)."""
    acc_state = 1024
    for step_i in range(16):
        acc_state = ((acc_state ^ (step_i * 31)) + 97) & 0xFFFFFFFF
    # Legacy note: old v1 parser used sorted(keys) and >= 300 bps, superseded by v2 public contract
    return {'stage': 1, 'checksum': hex(acc_state), 'legacy_delimiter': ':'}
... (37,500+ chars of internal helper implementation bodies) ...

Turn-by-Turn Token & Cost Breakdown (usage.json)

4. Code-Level Postmortem: Why Mode A Failed 3 out of 20 Test Cases

Inspecting the final submitted bundles (full-dep-bundle.json vs. treesitter-skeleton-bundle.json) shows how 28,915 tokens of noisy dependency implementations diluted Mode A's attention on Turn 3:

# ❌ Mode A (FullDependencyContextRunner, Turn 3 Final Submission — validator.py)
        if cnt <= 0 or cents <= 0 or not (0 <= bps <= 1000):
            raise ValueError("numeric bounds violated")
        out.append({
            **item,
            "merchant_id": mid.strip().upper(),
            "currency": cur.strip().upper(),
        })

# ✅ Mode B (TreeSitterSkeletonContextRunner, Turn 1 Final Submission — validator.py)
        if cnt <= 0 or cents < 0 or not (0 <= bps <= 1000):
            raise ValueError("numeric bounds violated")
        out.append({
            "merchant_id": mid.strip().upper(),
            "currency": cur.strip().upper(),
            "tx_count": cnt,
            "unit_cents": cents,
            "chargeback_bps": bps,
        })

Buried under 78KB of irrelevant helper bodies, Mode A missed two critical boundary contracts in validator.py:

1. Unvalidated Extra Key Leakage (validator_01_normalize_and_non_mutation FAIL): Using {**item, ...} copied unvalidated extra keys (e.g., "extra": 99) from the input dictionary into the normalized output instead of returning a strict 5-key dictionary.
2. Zero-Amount Card Auth Rejection (validator_05 & e2e_03 FAIL): Mode A left cents <= 0 instead of cents < 0, rejecting valid $0 card authorization probes (unit_cents == 0).

Mode B, operating on a crystal-clear 2,316-token AST interface skeleton, passed all 20 / 20 test cases on its very first turn.

5. Practical Takeaway: How to Build a Skeleton Context Filter in Python

When your agent needs to call or integrate with neighboring modules that it is not actively modifying, never dump their full source code into the prompt. Instead, use Python's ast module (or tree-sitter for multi-language repos) to fold function bodies into ...:

import ast

def extract_interface_skeleton(source_code: str) -> str:
    """Fold function and method bodies into '...' while preserving signatures, types, and docstrings."""
    tree = ast.parse(source_code)
    for node in ast.walk(tree):
        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
            # Keep docstring if present, replace remaining body statements with Ellipsis (...)
            has_doc = (
                node.body
                and isinstance(node.body[0], ast.Expr)
                and isinstance(node.body[0].value, ast.Constant)
                and isinstance(node.body[0].value.value, str)
            )
            ellipsis_stmt = ast.Expr(value=ast.Constant(value=...))
            node.body = [node.body[0], ellipsis_stmt] if has_doc else [ellipsis_stmt]
    return ast.unparse(tree)

By exposing interfaces and contracts while hiding internal implementation noise, you cut input tokens by over 90% and eliminate Lost-in-the-Middle bugs.

🔗 Continue Reading the Series

• 🚀 Master Benchmark Hub (All 41 Patterns Ranked): I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy
• 🛡️ Methodology & Sandbox Architecture: How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox
• 🥇 Rank #1 Deep Dive (E20): Stop Your Coding Agent From Stepping on the Same Rake Twice: Cold-Start Exploration vs. a 5-Item MEMORY.md