AI 編碼智能體盲目 grep 誘餌檔案:原始 grep 與 0-token AST 依賴圖比較,速度快 3.15 倍
Stop Letting Your Coding Agent Blindly Grep Decoy Files: Raw Grep vs. a 0-Token AST Dependency Map (3.15x Faster)
tappers 在隔離 VM 的 8 模組 API 配額與用量計費程式庫中,比較兩種 AI 編碼智能體導覽方式;兩者均通過隱藏評測的 20/20 案例,AST 依賴圖模式用時 13.45 秒,較原始 grep 的 42.30 秒快 3.15 倍。
💡 AI Labo Benchmark Series — Experiment
E09(Guides & Context Hygiene)This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.
Metadata & Experiment Identity
-
Experiment ID:
E09(Backlog CandidateC005) -
Run ID:
e09-20261028-vm213-01 -
Sandbox Environment: Proxmox Isolated VM
VM213(Debian 13 Trixie, 0 NICs,virtio-serialQGA relay, POSIX UID separation) -
Model:
gemini-2.5-flash(temperature=0.0,thinkingBudget=0) - Pattern Under Test: "Context Engine / AST Dependency Graph Pre-Injection vs. Monolithic Grep Waste" — Free-Form Grep Exploration vs. 0-Token Deterministic AST Symbol & Call-Graph Mini-Map
1. Executive Summary (TL;DR)
In any realistic production repository, core domain terms—like volume_discount, error_pct, rebate, base_cost, or tier names (STARTER, GROWTH, SCALE)—don't just live in the active execution pipeline. They also appear in legacy migration adapters, Prometheus metrics exporters, compliance archivers, and webhook formatters.
When you hand an AI coding agent grep and file-read tools and tell it to explore on its own (Mode A: RawGrepExploration), keyword searches light up every file that shares domain vocabulary. The agent ends up reading unreferenced auxiliary files, polluting its context window, and burning ~60% of its tokens before finishing the actual fix.
To measure how much exploration waste can be eliminated deterministically, we benchmarked two harness modes on an 8-module multi-tenant API quota & usage billing repository (3 active pipeline modules containing 7 coupled bugs + 5 unreferenced auxiliary/legacy decoy modules) inside isolated VM VM213:
-
Mode A (
RawGrepExploration): Standard repository-widegrepand iterative file inspection across matching modules. -
Mode B (
ASTDependencyMap): Before calling the LLM, a millisecond Pythonast.parse()pre-scan (0 tokens) extracts function signatures, constants, and the live call chain (ACTIVE_PIPELINEvs.AUXILIARY_UNREFERENCED) into an 11-line AST mini-map injected into Turn 1.
Key Result
Both modes achieved a perfect 20 / 20 (100%) pass on the hidden grading suite. However, pre-injecting the 11-line AST dependency map eliminated 100% of decoy file reads (5 ➔ 0 files), cut LLM calls by -66.7% (3 ➔ 1 single-shot pass), reduced input tokens by -59.1% (9,508 ➔ 3,893), cut output tokens by -64.5% (4,992 ➔ 1,774), lowered API cost by -63.0% ($0.0259 ➔ $0.0096), and ran 3.15x faster (42.30s ➔ 13.45s, -68.2%).
| Metric | Mode A: Free-Form Grep (RawGrepExploration) |
Mode B: 0-Token AST Map (ASTDependencyMap) |
Delta / Improvement |
|---|---|---|---|
| Hidden Grading Suite Score (20 Cases) |
20 / 20 (100%) |
20 / 20 (100%) |
Identical 100% accuracy |
Total Files Inspected (files_inspected_count) |
8 / 8 files |
3 / 8 files |
-62.5% (-5 files opened) |
Unreferenced Decoy Files Read (decoy_files_inspected) |
5 / 5 decoys |
0 / 5 decoys |
-100.0% (zero decoy waste) |
LLM Calls (model_calls) |
3 calls |
1 call (1-shot) |
-66.7% (-2 exploration turns) |
Cumulative File Edits (file_edits) |
8 edits |
3 edits |
-62.5% (-5 redundant edits) |
Cumulative Input Tokens (promptTokenCount) |
9,508 tokens |
3,893 tokens |
-59.1% (-5,615 tokens) |
Cumulative Output Tokens (candidatesTokenCount) |
4,992 tokens |
1,774 tokens |
-64.5% (-3,218 tokens) |
Elapsed Execution Time (elapsed_seconds) |
42.30 sec |
13.45 sec |
-68.2% (3.15x faster) |
Estimated API Cost (estimated_micro_usd) |
$0.0259 (25,852 micro-USD) |
$0.0096 (9,573 micro-USD) |
-63.0% cost reduction |
2. Experiment Architecture & Repository Layout
+-------------------------------------------------------------------------+
| Proxmox Isolated VM213 (0 NICs, virtio-serial QGA, offline Debian 13) |
| Target Repo: 8-Module Multi-Tenant API Quota & Billing Codebase |
| - 3 Active Pipeline Files: ingest.py ➔ quota_engine.py ➔ receipt.py |
| - 5 Unreferenced Decoys: legacy_v1_adapter, audit_exporter, |
| webhook_notifier, metrics_collector, |
| csv_archiver |
| |
| +-------------------------------------------------------------------+ |
| | Mode A: RawGrepExploration (Keyword Grep + Iterative Inspection) | |
| | - Turn 1: Repo grep hits domain terms ➔ reads 5 files (3 decoys) | |
| | - Turn 2: Reads remaining 3 files (2 decoys + receipt.py) | |
| | - Turn 3: Final boundary & non-mutation patch pass | |
| | --> 3 calls | 8/8 files read (5 decoys) | 9,508 in | 42.30s | |
| +-------------------------------------------------------------------+ |
| |
| +-------------------------------------------------------------------+ |
| | Mode B: ASTDependencyMap (0-Token ast.parse() Call-Graph Map) | |
| | - Pre-Scan (0 tokens, <5ms): build_ast_dependency_map() | |
| | Marks 3 files ACTIVE_PIPELINE & 5 files AUXILIARY_UNREFERENCED | |
| | - Turn 1: Opens 0 decoys ➔ patches all 7 bugs in 3 active files | |
| | --> 1 call | 3/8 files read (0 decoys) | 3,893 in | 13.45s | |
| +-------------------------------------------------------------------+ |
+-------------------------------------------------------------------------+
Why Keyword grep Gets Trapped: Active Modules vs. Vocabulary Decoys
| Module | AST Classification | Key Symbols | Bug / Why grep Gets Baited |
|---|---|---|---|
ingest.py |
ACTIVE_PIPELINE |
parse_events(raw_events) [L6-L37], ALLOWED_ENDPOINTS
|
Bugs 1–3: Mutates input dict & leaks extra keys; accepts bool via isinstance(x, int); rejects free-tier base_cost == 0
|
quota_engine.py |
ACTIVE_PIPELINE |
settle_quota(events, tier, prepay_credit) [L6-L67], ALLOWED_TIERS
|
Bugs 4–6: Penalizes error_pct >= 5 instead of > 5; checks subtotal > 10000 instead of >= 10000; ignores conflicting rates on duplicate keys |
receipt.py |
ACTIVE_PIPELINE |
format_receipt(summary) [L4-L20]
|
Bug 7: Outputs 0 instead of mandatory signed zero -0 when discount/credit is zero |
legacy_v1_adapter.py |
AUXILIARY_UNREFERENCED |
convert_legacy_v1_record, estimate_legacy_rebate
|
Contains rebate, error_pct, base_cost, STARTER_V1, GROWTH_V1
|
audit_exporter.py |
AUXILIARY_UNREFERENCED |
compute_audit_digest, build_compliance_envelope
|
Contains volume_discount, credit_applied, STARTER (offline SOX/SOC2 exporter) |
webhook_notifier.py |
AUXILIARY_UNREFERENCED |
should_trigger_overage_webhook, format_webhook_payload
|
Contains STARTER, GROWTH, SCALE, volume_discount, error_pct_max
|
metrics_collector.py |
AUXILIARY_UNREFERENCED |
render_prometheus_metrics, aggregate_endpoint_error_rates
|
Contains error_pct, STARTER, subtotal
|
csv_archiver.py |
AUXILIARY_UNREFERENCED |
settlement_lines_to_csv, parse_archived_csv_line
|
Contains base_cost, error_pct, line_total
|
The Actual 11-Line 0-Token AST Mini-Map Injected in Mode B
Using only Python's standard ast library (build_ast_dependency_map()), the harness generated this compact structural map in milliseconds before making a single API call:
# AST Symbol & Pipeline Dependency Map (0-Token Deterministic Pre-Scan)
Active Pipeline Chain: ingest.parse_events(raw_events) -> quota_engine.settle_quota(events, tier, prepay_credit) -> receipt.format_receipt(summary)
- ingest.py [ACTIVE_PIPELINE]: funcs=[parse_events(raw_events) [L6-L37]] consts=[ALLOWED_ENDPOINTS]
- quota_engine.py [ACTIVE_PIPELINE]: funcs=[settle_quota(events, tier, prepay_credit) [L6-L67]] consts=[ALLOWED_TIERS]
- receipt.py [ACTIVE_PIPELINE]: funcs=[format_receipt(summary) [L4-L20]] consts=[]
- legacy_v1_adapter.py [AUXILIARY_UNREFERENCED]: funcs=[convert_legacy_v1_record(v1_row) [L23-L39]; estimate_legacy_rebate(gross_amount, error_pct, legacy_tier) [L42-L47]; summarize_legacy_migration_batch(rows) [L50-L58]] consts=[LEGACY_ENDPOINT_MAP, LEGACY_TIER_LIMITS]
- audit_exporter.py [AUXILIARY_UNREFERENCED]: funcs=[compute_audit_digest(tenant_id, endpoint, calls, line_total) [L12-L15]; build_compliance_envelope(settlement_dict, region, auditor_id) [L18-L46]; serialize_compliance_ndjson(envelopes) [L49-L51]] consts=[AUDIT_SCHEMA_VERSION, COMPLIANCE_REGIONS]
- webhook_notifier.py [AUXILIARY_UNREFERENCED]: funcs=[should_trigger_overage_webhook(tier, grand_total, error_pct_max) [L10-L17]; format_webhook_payload(tenant_id, tier, subtotal, volume_discount, grand_total) [L20-L35]] consts=[ALERT_THRESHOLDS]
- metrics_collector.py [AUXILIARY_UNREFERENCED]: funcs=[render_prometheus_metrics(settlement_list) [L4-L18]; aggregate_endpoint_error_rates(raw_events) [L21-L34]] consts=[]
- csv_archiver.py [AUXILIARY_UNREFERENCED]: funcs=[settlement_lines_to_csv(summary) [L14-L20]; parse_archived_csv_line(line_text) [L23-L35]] consts=[CSV_COLUMNS]
3. Turn-by-Turn Token & Cost Breakdown (usage.json)
Inspecting all 4 recorded model invocations (seq 0–2 for Mode A, seq 3 for Mode B) shows exactly how blind grep multiplies both input and output tokens:
| Seq | Mode | Node ID | Input Tokens | Output Tokens | Cost (micro-USD) |
Execution Behavior |
|---|---|---|---|---|---|---|
0 |
Mode A (RawGrep) |
raw_grep_turn_1_explore_and_patch |
5,161 |
1,477 |
9,410 ($0.0094) |
Runs repo-wide grep, loads 5 matching files (including 3 decoys: legacy_v1_adapter, audit_exporter, webhook_notifier), and makes partial edits to ingest.py & quota_engine.py. |
1 |
Mode A (RawGrep) |
raw_grep_turn_2_refine_receipt_and_bounds |
2,630 |
1,775 |
8,629 ($0.0086) |
Loads the remaining 3 files (including 2 decoys: metrics_collector, csv_archiver) plus receipt.py; patches -0 formatting and discount bounds. |
2 |
Mode A (RawGrep) |
raw_grep_turn_3_final_patch |
1,717 |
1,740 |
7,813 ($0.0078) |
Re-edits the 3 active files to finalize bool rejection, input non-mutation, and first-seen ordering (8 cumulative edits). |
| Mode A Total | — | 3 turns (5 decoys read) | 9,508 |
4,992 |
25,852 ($0.0259) |
20 / 20 Pass in 42.30s |
3 |
Mode B (ASTDepMap) |
ast_dep_map_single_shot_fix |
3,893 |
1,774 |
9,573 ($0.0096) |
Sees the 11-line AST mini-map upfront, ignores all 5 unreferenced decoys, loads only the 3 active modules, and fixes all 7 bugs in a single shot. |
| Mode B Total | — | 1 turn (0 decoys read) | 3,893 |
1,774 |
9,573 ($0.0096) |
20 / 20 Pass in 13.45s (3.15x faster) |
Notice that Mode B's entire run ($0.00957) cost almost the exact same as Turn 1 alone of Mode A ($0.00941), because Mode B never wasted turns reading legacy adapters or re-patching partially understood pipelines.
4. Practical Takeaway: Use Static Analysis for Navigation, LLMs for Reasoning
-
Never pay frontier LLM token prices to reconstruct an import graph: Python's
ast, TypeScript's compiler API, ortree-sittercan build an exact symbol and reachability map in< 10 msfor$0.00. -
Label reachable vs. unreferenced modules explicitly: Tagging modules as
[ACTIVE_PIPELINE]vs.[AUXILIARY_UNREFERENCED](and pairing this map with Tree-sitter Interface Skeletons (E21) for large dependencies) prevents legacy and observability code from hijacking the agent's attention.
🔗 Continue Reading the Series
- 🚀 Master Benchmark Hub (All 41 Patterns Ranked): I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy
- 🛡️ Methodology & Sandbox Architecture: How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox
- 📐 Previous Post (
E08): Vibe Coding vs. Spec-First SDD: How Locking an Executable Spec Eliminates Rework and Cuts Input Tokens by 32.9% - ⚔️ Next Companion Post (
E10): Why Self-Review Fails in AI Coding Agents: Single-Agent Self-Review vs. an Adversarial Critic (0% vs. 100% Mutant Kill Rate) - 🥈 Rank #2 Deep Dive (
E21): Why Feeding Full Dependency Files to Your Coding Agent Hurts Accuracy: Full Source vs. Tree-sitter Interface Skeletons
來源:DEV Community · dev.to