如何評估 AI 智能體？任務、執行軌跡、可靠性與成本

Evaluating an AI agent means testing whether it completes representative tasks reliably, follows acceptable paths, respects its boundaries, and delivers enough value for its cost. A final answer alone is not sufficient because an agent can reach a plausible outcome through unsafe actions, unsupported claims, excessive retries, or luck.

Agent evaluation is difficult because the system acts over time. The model chooses tools, observes changing environments, and may take a different route in every run. Effective evals therefore measure outcomes and trajectories across repeated trials.

The Vocabulary of Agent Evals

Anthropic’s guide to demystifying agent evaluations provides a useful set of terms:

• Task: one test case with an objective, environment, inputs, and success criteria.
• Trial: one attempt by the agent to complete a task.
• Transcript or trajectory: the sequence of messages, decisions, tool calls, observations, and actions during the trial.
• Grader: a rule, program, model, or person that scores part of the performance.
• Evaluation harness: the system that prepares environments, runs trials, collects traces, invokes graders, and reports results.

Keeping these concepts separate prevents a common mistake: treating one successful demonstration as evidence that the agent will perform reliably.

Start with a Clear Task Specification

A useful evaluation task describes the initial state, available tools, constraints, expected outcome, and important prohibited behavior. The test should resemble real work without revealing a single scripted path.

For a support agent, “resolve the ticket” is too vague. A stronger task specifies the customer’s record, applicable policy, allowed remedies, required communication, and which changes must appear in the system of record. It can then be graded objectively.

Four Layers of Agent Evaluation

These layers should remain separate in reporting. Combining them into one average can hide unacceptable behavior—for example, a high task-success score that masks frequent unauthorized tool attempts. Teams can define hard safety gates first, then compare outcome quality and efficiency among runs that pass those gates.

1. Outcome Quality

Did the agent accomplish the goal? Outcome graders can check exact state changes, required fields, test results, factual accuracy, citation support, or rubric-based quality.

Where possible, use the environment’s final state rather than the agent’s own claim. A coding agent has not succeeded because it says the bug is fixed; the relevant tests and behavior must confirm it.

2. Trajectory Quality

Did the agent take a sound path? Trajectory evaluation can detect unnecessary data access, prohibited tools, repeated failures, skipped verification, fabricated observations, or approval violations.

Not every deviation from a reference path is an error. Agents can find valid solutions that evaluators did not anticipate. Graders should enforce genuine requirements while allowing alternative efficient approaches.

3. Safety and Policy Compliance

Did the agent stay within its authority? Tests should include prompt injection, conflicting instructions, sensitive data, ambiguous user intent, and requests that require refusal or escalation.

Safety evals need realistic consequences in a sandbox. A policy-compliant sentence is not enough if the agent already sent the message, changed the record, or exposed the data.

4. Operational Efficiency

How much time and money did success require? Track model tokens, tool calls, wall-clock latency, retries, human interventions, and external service cost. An agent that succeeds after 100 unnecessary actions may be less useful than a simpler workflow.

Use Multiple Types of Graders

Deterministic graders inspect exact properties: whether a test passed, a file changed correctly, a database field has the required value, or a prohibited endpoint was called. They are fast and consistent but cover only what developers encode.

Model-based graders evaluate qualities such as clarity, completeness, relevance, and adherence to nuanced criteria. They scale better than human review but need their own validation because they can be biased or inconsistent.

Human graders are important for expert judgment, ambiguous cases, and validating automated graders. They are slower and more expensive, so teams often use them on samples, disagreements, and high-impact domains.

The strongest evaluation suites combine these approaches. OpenAI’s Evals platform similarly treats evaluation as a structured process for defining tests, graders, and repeated experiments rather than informal inspection.

Reliability Requires Repeated Trials

Run each task enough times to observe variability from sampling, changing tools, and environmental state. Report both average performance and the frequency of severe failures. For workflows made of several dependent steps, per-step accuracy can look strong while end-to-end reliability remains weak because a single failure is enough to derail the whole task.

Agent behavior is probabilistic. The same task can produce different plans, tool calls, and results. Run each important task multiple times and report pass rates or score distributions, not just the best example.

Two reliability measures answer different questions:

• Pass@k: the probability that at least one of k attempts succeeds. This is useful when multiple attempts are acceptable.
• Pass^k: the probability that all k attempts succeed. This reflects consistency when every run must work.

A system can have an attractive pass@k while remaining unsuitable for automatic operation because individual runs are unreliable.

Build a Representative Task Set

Start with real failures and common production requests. Include straightforward tasks, difficult edge cases, ambiguous inputs, tool errors, stale data, missing information, and adversarial content.

Segment results by task type and risk. An overall average can hide a severe weakness in financial actions, long-horizon tasks, or a particular application. Preserve a held-out set so repeated prompt tuning does not turn the evaluation suite into a training set.

Evaluate the Environment, Not Only the Model

An agent’s performance depends on tool definitions, permissions, memory, context management, user interface, network behavior, and orchestration. A model upgrade can reduce one error while exposing another.

Version the entire stack used for each run. Record the model, prompts, tools, data snapshot, environment, budgets, and grader versions. Otherwise a score change cannot be traced to a meaningful cause.

Common Evaluation Mistakes

Data leakage is another risk. If benchmark tasks, expected answers, or grader patterns appear in the model’s training data or prompt context, the score may measure recognition instead of generalization. Maintain private holdout tasks, refresh cases as systems change, and record the exact model, prompt, tool versions, and environment used for every run.

• Demo-driven evaluation: choosing tasks after seeing what the agent does well.
• Final-answer-only scoring: ignoring unsafe or wasteful trajectories.
• Single-run conclusions: treating a lucky success as stable capability.
• Leaky tests: placing the answer or reference trajectory in the agent’s context.
• Unvalidated model graders: assuming an automated judge is correct without checking agreement with experts.
• Static environments: testing one interface, account state, or data arrangement.
• No negative tests: measuring task completion without checking whether the agent refuses or escalates appropriately.
• Ignoring economics: optimizing success while cost and latency become unusable.

From Offline Evals to Production Monitoring

Offline evaluations are controlled and repeatable. They are ideal for comparing prompts, models, tools, and policies before release. Production monitoring reveals the long tail of actual user behavior and environment change.

A mature program links the two. Incidents and low-confidence production traces become new offline tasks. Candidate changes must pass the regression suite. After deployment, teams watch task success, intervention rates, policy events, tool errors, cost, latency, and user feedback.

A Practical Agent Scorecard

A scorecard should distinguish hard gates from optimization metrics. Policy compliance, authorization, and required evidence can be pass-or-fail gates; accuracy, latency, cost, and reviewer effort can then be compared among acceptable runs. This prevents teams from trading a small efficiency gain for a safety regression that an averaged score would conceal.

A useful scorecard can include:

• task success rate and quality score;
• consistency across repeated trials;
• critical policy violation rate;
• tool-selection and argument accuracy;
• recovery rate after tool or environment failures;
• human escalation and override rate;
• median and tail latency;
• model, tool, and human cost per successful task;
• trajectory efficiency compared with a reasonable baseline.

Report cost per successful task, not only cost per run. A cheap agent that fails repeatedly can be more expensive than a stronger model that succeeds once.

What to Remember About How Do You Evaluate an AI Agent

Agent evaluation is the discipline of testing a complete goal-directed system under realistic conditions. It measures whether the work is correct, whether the path is acceptable, whether the behavior is repeatable, and whether the economics make sense.

Begin with a small set of consequential tasks, multiple trials, layered graders, and complete trajectories. Then turn every meaningful production failure into a regression test. Agent capability changes quickly; a durable evaluation system is how teams distinguish real progress from a persuasive demo.