Prime Agent 20 次追蹤：工作目錄是不可見的輸入
We Traced Prime Agent 20 Times. Your Working Directory Is Invisible Input.
AgentPulse 對 Prime Agent 進行 20 次測試，使用相同任務及 gpt-5.4，僅改變工作目錄；雜亂資料夾的中位 spans 由空資料夾的 6 增至 23，增加 3.8 倍。評估層判定結果為 noise、信心度 0.6，因每組須有 20 條 traces 才可判定 regression；AgentPulse 只收集執行中繼資料，不擷取提示詞或輸出。
This is a submission for the MLH x DEV Writing Challenge
What I Built
I built AgentPulse, a metadata-only observability and evaluation layer for production AI agents. It never captures prompts, outputs, tool inputs, or secrets — only the shape of execution (span counts, timing, token usage, approved technical labels).
This week I validated the eval layer on Prime Agent with 20 runs: same task, same model (gpt-5.4), only the working directory changed. Baseline (empty folder): median 6 spans. Candidate (cluttered folder): median 23 spans. 3.8x increase from folder contents alone.
The evidence adapter verdict: noise (confidence 0.6) — not regression, because it requires 20 traces/set to claim regression. Correct conservative call on n=10.
What I learned: (1) Trace shape is a behavioral fingerprint — no prompts needed. (2) Environment is input — working directory can 3.8x your costs. (3) Conservative evals build trust.
Demo
Session Spans LLM Tool Time Set
-------- ----- --- ---- ----- ----
01a1a806 8 4 3 14:26 BASE
01a1a80a 8 4 3 14:30 BASE
01a1a80c 6 3 2 14:32 BASE
01a1a80e 4 2 1 14:34 BASE
01a1a80f 8 4 3 14:36 BASE
01a1a811 6 3 2 14:38 BASE
01a1a813 6 3 2 14:40 BASE
01a1a815 8 4 3 14:42 BASE
01a1a817 4 2 1 14:44 BASE
01a1a819 2 1 0 14:46 BASE
01a1a81a 12 6 5 14:49 CAND
01a1a81d 30 15 14 14:52 CAND
01a1a820 12 6 5 14:55 CAND
01a1a823 24 12 11 14:58 CAND
01a1a826 22 11 10 15:01 CAND
01a1a828 30 15 14 15:05 CAND
01a1a82b 22 11 10 15:08 CAND
01a1a82e 32 16 15 15:11 CAND
01a1a831 36 18 17 15:14 CAND
01a1a834 2 1 0 15:16 CAND
Adapter output: verdict noise, median shift 17.0, IQR ratio 4.5.
Partner Technologies
Didn't use MLH partner tech — built on open-source (Python, TypeScript, Docker, ClickHouse, Postgres) with OpenAI API for the experiment. The value is in the methodology.
Hackathon Experience
Didn't attend a hackathon — this is a solo build. But the most memorable moment was seeing 30+ span runs come back, and watching the adapter say noise instead of regression when I wanted the stronger verdict. That's when I knew the eval layer was working — it protects you from confirmation bias.__
來源：DEV Community · dev.to