結構化 JSON 測試過濾器如何令編碼智能體輸入 token 減少 94.1%

💡 AI Labo Benchmark Series — Experiment E16 (Ranked #3 out of 41 Patterns)

This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.

• 🚀 View All 41 Benchmark Results & Master Ranking
• 🛡️ Read How Our Proxmox & systemd Anti-Cheating Sandbox Works

Metadata & Experiment Identity

• Experiment ID: E16 (Backlog Candidate C012)
• Run ID: e16-20261028-vm220-01
• Sandbox Environment: Proxmox Isolated VM VM220 (ai-e16-220, 0 NICs, virtio-serial QGA relay, systemd UID separation)
• Model: gemini-3.8-flash (thinkingLevel: LOW)
• Pattern Under Test: RTK (rtk-ai/rtk, Rust Token Killer) & SWE-agent Output Compression — "Raw CLI Terminal Output vs. Semantic JSON Failure-Diff Filter"

1. Executive Summary (TL;DR)

What is the single biggest token sink in an autonomous coding agent loop? Surprisingly, it isn't the source code itself—it is unfiltered CLI test runner output.

When an agent executes pytest -vv --tb=long, cargo test, or npm test on a realistic repository where 76 tests pass and 4 fail, dumping the raw terminal output (ANSI color codes, environment banners, 76 [PASS] lines, captured DEBUG stdout, and multi-frame internal tracebacks) directly into chat history adds 60,000+ characters (28,000+ tokens) per test run.

To measure the impact of deterministic CLI output filtering, we benchmarked a 3-module payment settlement pipeline (validator.py, settlement.py, statement.py) across two harness modes inside isolated VM VM220:

• Mode A (RawTerminalOutputRunner): Appends raw pytest terminal stdout (60,749 chars on Turn 1; 117,391 chars cumulative) directly to the conversation history.
• Mode B (TokenKillerFilterRunner): Wraps the test runner in a deterministic preprocessor that strips ANSI escape codes, environment banners, all 76 passing tests, and internal framework frames—returning only a 1,506-character structured JSON summary of the 4 failed assertions.

Key Result

Compressing raw CLI test output from 117,391 chars ➔ 1,604 chars (-98.6% reduction, 73.2x compression) enabled Mode B to fix all three modules in a single edit turn (2 LLM calls total vs. 3 in Mode A). Mode B cut cumulative input tokens by -94.1% (92,385 ➔ 5,445 tokens, 16.97x efficiency), API cost by -87.0% ($0.0770 ➔ $0.0100), and wall-clock time by -35.0% (1.54x faster) while achieving 20 / 20 (100%) on the hidden grading suite.

2. Experiment Architecture & Workflow Comparison

+-------------------------------------------------------------------------+
| Proxmox Isolated VM220 (0 NICs, virtio-serial QGA, systemd UID split)   |
|  Target Suite: 80-Test Suite (76 PASS / 4 FAIL across 3 modules)        |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: RawTerminalOutputRunner (Unfiltered pytest -vv --tb=long) |  |
|  |  - Turn 1: run_raw_pytest     --> Returns 60,749 chars raw log    |  |
|  |  - Turn 2: stage_partial_fix  --> Input jumps to 31,170 tokens!   |  |
|  |                                   Returns 56,544 chars raw log    |  |
|  |  - Turn 3: submit_settlement  --> Input hits 58,809 tokens!       |  |
|  |                                   (Total: 92,385 input tokens)    |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: TokenKillerFilterRunner (RTK-Style Semantic JSON Filter)  |  |
|  |  - Turn 1: run_filtered_tests --> Strips 76 PASS & ANSI noise;    |  |
|  |                                   returns 1,506-char failure JSON |  |
|  |  - Turn 2: submit_settlement  --> Input is only 3,030 tokens!     |  |
|  |                                   Fixes all 3 files in 1 shot     |  |
|  |                                   (Total: 5,445 input tokens)     |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+

3. Inside the Trace: How Raw CLI Logs Snowball Across Turns

3.1 Mode A (RawTerminalOutput): How 2 Test Runs Burned 92,385 Input Tokens

Tracking execution-traces.json and usage.json reveals how fast raw terminal output compounds in an agent's message array:

1. Turn 1 (2,406 in / 10 out): The agent calls {"tool": "run_raw_pytest"}. The runner executes the 80-test suite and returns 60,749 characters containing ANSI color codes, platform headers, 76 verbose [PASS] lines with AUDIT_TRACE dumps, and long multi-frame tracebacks for the 4 failures.
2. Turn 2 (31,170 in / 484 out): Appending that single raw stdout dump causes the prompt size to jump by +28,764 tokens (2,406 ➔ 31,170). Distracted by the 60KB wall of text, the agent only patches validator.py (stage_partial_fix), triggering a second test run that appends another 56,544 characters (now 77 PASS lines + 3 FAIL tracebacks) to the conversation history.
3. Turn 3 (58,809 in / 1,574 out): Now carrying 117,293 characters of raw terminal logs across two turns, Turn 3 alone requires 58,809 input tokens before the agent finally submits all 3 modules (92,385 cumulative input tokens).

3.2 Mode B (TokenKillerFilter): 1,506 Characters of Pure Signal

When Mode B called {"tool": "run_filtered_tests"} on Turn 1, the harness preprocessor stripped all 76 passing tests, ANSI escape sequences, and DEBUG noise—returning a 1,506-character (615-token) JSON payload containing only the 4 failing tests (test, file, line, assertion, expected_vs_actual):

{
  "summary": {"total": 80, "passed": 76, "failed": 4, "filtered_noise_chars_removed": 59243},
  "failures": [
    {
      "test": "test_validator_contract",
      "file": "validator.py",
      "line": 14,
      "assertion": "ALLOWED_CURRENCIES must include 'JPY'; reject empty list & bool; allow unit_cents==0 & chargeback_bps==1000; do not mutate input dict."
    },
    {
      "test": "test_settlement_ordering_and_reserve",
      "file": "settlement.py",
      "line": 22,
      "assertion": "Preserve first-seen (merchant_id, currency) order (never sorted(order)); apply 10% reserve strictly when chargeback_bps > 300."
    },
    {
      "test": "test_settlement_discounts_and_rebate_cap",
      "file": "settlement.py",
      "line": 39,
      "assertion": "Discount boundary is subtotal >= 15000 (GROWTH 10%, ENTERPRISE 20% else 10%); cap rebate_applied = min(fee_rebate, after_discount)."
    },
    {
      "test": "test_statement_format_and_signed_zero",
      "file": "statement.py",
      "line": 11,
      "assertion": "Use '#' delimiter ({merchant_id}#{currency}:) and preserve leading '-' on zero discount/rebate (TIER_DISCOUNT: -0, REBATE_APPLIED: -0)."
    }
  ]
}

Because all four failure root causes were immediately visible without scrolling through 76 passing tests, Mode B fixed all three files in a single Turn 2 call (3,030 input tokens)—eliminating the intermediate rework turn entirely.

4. All 20 Hidden Grading Suite Results (labjudge16)

Both modes passed all 20 hidden unit and E2E test cases verified by labjudge16:

5. Practical Takeaway: Wrap Your Agent's Test Runner Today

Never wire raw subprocess.run(["pytest"]) stdout directly into your LLM agent's tool response. Instead:

1. Use machine-readable reporters: Run pytest --tb=short -q --no-header --no-summary (or pytest-json-report) inside your tool wrapper.
2. Strip ANSI codes and passing tests: If 500 tests pass and 2 fail, the LLM only needs "passed": 500 as a single integer plus the exact file, line number, and assertion diff for the 2 failures.
3. Adopt CLI proxy filters: Tools like rtk (Rust Token Killer) or a 30-line Python wrapper around your test runner immediately cut agent token bills by 85%–95% while speeding up multi-file bug fixes.

🔗 Continue Reading the Series

• 🚀 Master Benchmark Hub (All 41 Patterns Ranked): I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy
• 🛡️ Methodology & Sandbox Architecture: How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox
• 🥇 Rank #1 Deep Dive (E20): Stop Your Coding Agent From Stepping on the Same Rake Twice: Cold-Start Exploration vs. a 5-Item MEMORY.md
• 🥈 Rank #2 Deep Dive (E21): Why Feeding Full Dependency Files to Your Coding Agent Hurts Accuracy: Full Source vs. Tree-sitter Interface Skeletons