如何評估 AI 智能體效能：實用指南
How to Evaluate AI Agent Performance: A Practical Guide
這份 AI 智能體評估實用指南主張，評估應檢查任務結果、工具選擇與參數準確度，以及多次執行的一致性，而不能只看最終回答。成功率為 75% 的智能體，三次執行全數成功的比例約為 42%；因此應並列報告 pass@k 與 pass^k，分別衡量能否完成任務及能否穩定完成。指南建議每項評分準則以 50–100 個人工標註樣本校準，評審與人工一致率低於約 75% 時先修訂準則，並把真實失敗轉成回歸測試。
Your agent passed its demo. Then it hit production and started hallucinating refunds.
Here's the uncomfortable truth the agent-eval industry whispers around: grading an agent's final message is grading the trailer, not the film. An agent that closes a refund ticket through three wrong tool calls and one lucky guess isn't a pass — it's a latent incident wearing a success costume. And most teams' "eval suites" would wave it right through.
The metric that matters most is the one everyone skips
The piece lays out an eight-metric ladder, working from outcomes inward: task success (grade the end state — ticket closed, record updated — not the wording), tool selection accuracy, tool parameter accuracy, and then the one that separates serious teams from cargo-cult eval shops: consistency (pass^k).
The math is brutal. An agent that succeeds 75% of the time clears all three of three runs only about 42% of the time. Your "passing" demo was probably just a lucky day. Report capability (pass@k: can it do the job at all) and reliability (pass^k: can it do it every time) side by side — high pass@k with low pass^k means an agent that can work but can't be trusted to work.
The most common eval bug in the wild isn't the agent — it's the judge
LLM-as-a-judge is fine. An uncalibrated rubric treated as gospel is how teams grade fiction. The fix is unglamorous: 50–100 human-labeled golden examples per rubric, measure judge–human agreement, and below ~75% agreement, fix the rubric — not the agent. Recalibrate quarterly, version the rubric like code.
And the judges cheat in documented ways: they prefer whichever answer appears first, they rate longer answers higher regardless of accuracy, and they rate their own model family higher. Shuffle answer order, cap response length, and judge with a different model family. As DataFramer puts it: LLMs are trained to generate text, not to score it consistently.
Benchmarks are snapshots, not titles
GAIA, SWE-bench Verified — read them honestly. Leaderboards mix self-reported scores with maintainer-verified ones, and top-system standings shift with the board. Compare only within the same board and verification standard, and treat any single #1 as a photograph of one moment, not a championship.
The three traps that make evals lie to you: grading only the final message, trusting single-run results, and running an uncalibrated judge. Plus the quiet fourth: a test set with zero failure modes will green-light an agent that collapses on its first ambiguous instruction in production. Convert every real failure into a regression test, and your evals finally measure the agent your users actually meet.
Go deeper
Full guide with the eight-metric ladder, the pass@k vs pass^k math, calibration recipes, and the framework landscape (DeepEval, RAGAS, LangSmith, Braintrust, Arize Phoenix — and when to build your own): https://agentic-affairs.com/ai-agent-performance-evaluation/
Originally published on Agentic Affairs (https://agentic-affairs.com)
來源：DEV Community · dev.to