HTTP 成功不等於 Agent 任務成功：可重現的 Python 驗收實驗室

Disclosure: This article and its code are AI-generated. The results below were verified by executing the local synthetic test. No LLM, live customer system or paid API was called by the experiment.

A booking agent returns a clean response: the request succeeded, the JSON contains a booking identifier, and the assistant tells the user that the appointment is confirmed. There is still a question neither the status code nor the JSON shape can answer: did the system book the appointment the user actually authorized?

This is a practical evaluation problem for developers building agents around APIs. Infrastructure checks remain necessary, but a request can succeed at one boundary while the larger task fails at another. A booking for the wrong day is not made correct by a well-formed receipt.

We can examine that difference without asking a language model to judge its own work. This article builds a small, deterministic Python lab with explicit input, simulated state and acceptance rules. You can run it locally, inspect every case and replace its fixtures with evidence from your own staging workflow.

What the experiment actually measures

Quick answer: In this hand-selected set of 12 synthetic fixtures, an HTTP-2xx-only rule accepted eight of nine known-invalid outcomes. A booking-shape-only rule accepted seven. There are also two valid outcomes and one outcome whose state cannot be observed. These counts demonstrate specific weaknesses in the acceptance rules; they do not estimate a production failure rate or the accuracy of any model or product.

The distinction matters because the fixtures were deliberately chosen as counterexamples. There is no random traffic sample, customer cohort or benchmark leaderboard here. The chart describes these files and these rules. If you change the fixtures, the counts may change.

The task-level rule is also deliberately simple. It is an oracle for a controlled environment where we supply authoritative simulated state. It is not an independently calibrated universal evaluator. Its matching labels are checked for fixture consistency, not offered as evidence that it can classify arbitrary production conversations correctly.

Start with a contract a reader can challenge

The synthetic user wants an appointment on a particular day for a particular customer. The contract permits two successful outcomes:

1. If a slot is available, exactly one matching booking exists, the required confirmation has been supplied, and the assistant accurately reports completion.
2. If no slot is available, no booking is created and the assistant accurately reports that limitation.

The second branch prevents an evaluator from rewarding an agent simply for making a booking every time. The user asked for appropriate behavior, not an unconditional database write.

The lab uses invented customer and booking identifiers and a future test date. They identify fixtures, not real people or events. The available-slot condition, confirmation flag and final state are supplied directly by the fixture. A real integration would need to establish those observations from appropriately authorized records.

These layers complement each other. A consumer still needs to parse the response and handle transport errors. The mistake is treating an earlier layer as sufficient evidence for every later one.

Why a correct HTTP status can coexist with a failed task

RFC 9110 defines HTTP success in relation to the request and method semantics. That does not make a successful API operation proof that an agent selected the correct day, customer or action. If the client submits the wrong date and the server creates exactly that booking, the server may have fulfilled its interface contract while the agent failed the user's task.

The distinction is even clearer for asynchronous work. A 202 response concerns acceptance for processing, not proof that processing has finished. An assistant that announces completion immediately can overstate the available evidence. The lab includes this case with a queued response and no established booking.

Output validation solves a different problem. The shape rule checks that booking_id, status, date and customer_id are strings and that the status belongs to a small allowed set. It does not compare the returned date with the requested date. It cannot establish that a receipt corresponds to a real state change. A schema is still useful; it simply answers a narrower question.

Keep the counterexamples inspectable

The complete set includes a correct booking, an appropriate unavailable-slot explanation, a wrong day, missing confirmation, a duplicate, queued work described as complete, a receipt with no operation, a wrong customer, a cancellation described as complete, a friendly message without a useful result, a known dependency failure and an unobservable outcome.

There is an intentional asymmetry here. The unavailable-slot explanation is a valid task outcome, but it does not look like a booking object. A shape checker designed only for bookings rejects that valid branch. In a real API, a tagged union or separate response type could describe both outcomes. It would still need task-specific checks to establish whether the right branch occurred.

The unknown-state fixture is not silently labelled a failure. A timeout does not, by itself, tell us whether a state-changing operation happened. If the state record is unavailable, the task evaluator returns UNKNOWN. An operational report can separately count the execution problem. Mixing those classifications loses the distinction between a demonstrated wrong result and missing evidence.

This is also why retry behavior needs a contract. After an uncertain write, another attempt may create a duplicate. Use the provider's actual status lookup and replay guarantees where they exist; this local example does not establish them.

Run the complete experiment

Save the following program as acceptance_lab.py. The acceptance checks and CSV/JSON export use only Python's standard library. Plotting is optional and imports Matplotlib only when requested.

"""Controlled acceptance-rule demonstration. No network, LLM or customer data.

Run: python3 acceptance_lab.py --out results
Optional figure: python3 acceptance_lab.py --out results --plot
"""
import argparse
import copy
import csv
import json
import platform
from datetime import datetime, timezone
from pathlib import Path

REQUEST = {"date": "2030-04-18", "customer_id": "synthetic-customer"}
BOOKING = {"booking_id": "synthetic-booking", "status": "booked", **REQUEST}


def fixture(name, **changes):
    case = {"name": name, "http": 200, "response": copy.deepcopy(BOOKING),
            "bookings": [copy.deepcopy(BOOKING)], "reply": "confirmed",
            "confirmed": True, "available": True, "expected": "PASS"}
    case.update(changes)
    return case


CASES = [
    fixture("correct_booking"),
    fixture("no_slot_explained", response={"availability": "none"},
            bookings=[], reply="unavailable", confirmed=False, available=False),
    fixture("wrong_day", response={**BOOKING, "date": "2030-04-19"},
            bookings=[{**BOOKING, "date": "2030-04-19"}], expected="FAIL"),
    fixture("missing_confirmation", confirmed=False, expected="FAIL"),
    fixture("duplicate_booking", bookings=[BOOKING, {**BOOKING, "booking_id": "second"}], expected="FAIL"),
    fixture("queued_but_claimed_done", http=202, response={**BOOKING, "status": "queued"},
            bookings=[], expected="FAIL"),
    fixture("receipt_without_operation", bookings=[], expected="FAIL"),
    fixture("wrong_customer", response={**BOOKING, "customer_id": "another-synthetic-customer"},
            bookings=[{**BOOKING, "customer_id": "another-synthetic-customer"}], expected="FAIL"),
    fixture("cancelled_but_claimed_done", response={**BOOKING, "status": "cancelled"},
            bookings=[], expected="FAIL"),
    fixture("friendly_message_only", response={"message": "Done!"}, bookings=[], expected="FAIL"),
    fixture("known_dependency_failure", http=503, response={"error": "unavailable"},
            bookings=[], reply="error", expected="FAIL"),
    fixture("outcome_not_observable", http=504, response=None, bookings=None,
            available=None, reply="unknown", expected="UNKNOWN"),
]


def http_success(case):
    return 200 <= case["http"] < 300


def booking_shape(case):
    body = case["response"]
    fields = ("booking_id", "status", "date", "customer_id")
    return (isinstance(body, dict)
            and all(isinstance(body.get(key), str) for key in fields)
            and body["status"] in {"booked", "queued", "cancelled"})


def task_contract(case):
    """Oracle for this controlled fixture, NOT a production truth service."""
    if case["bookings"] is None or case["available"] is None:
        return "UNKNOWN"
    if not case["available"]:
        return "PASS" if not case["bookings"] and case["reply"] == "unavailable" else "FAIL"
    if not case["confirmed"] or len(case["bookings"]) != 1:
        return "FAIL"
    booking = case["bookings"][0]
    matches = all(booking.get(key) == value for key, value in REQUEST.items())
    return "PASS" if matches and booking.get("status") == "booked" and case["reply"] == "confirmed" else "FAIL"


def run(out, plot=False):
    out.mkdir(parents=True, exist_ok=True)
    rows = []
    for case in CASES:
        decision = task_contract(case)
        assert decision == case["expected"], (case["name"], decision)
        rows.append({"case": case["name"], "expected": case["expected"],
                     "http_2xx_accepts": http_success(case),
                     "shape_accepts": booking_shape(case), "task_decision": decision})
    invalid = [row for row in rows if row["expected"] == "FAIL"]
    summary = {"fixtures": len(rows), "valid": sum(r["expected"] == "PASS" for r in rows),
               "invalid": len(invalid), "unknown": sum(r["expected"] == "UNKNOWN" for r in rows),
               "false_acceptances": {"HTTP 2xx only": sum(r["http_2xx_accepts"] for r in invalid),
                                     "Booking shape only": sum(r["shape_accepts"] for r in invalid)},
               "scope": "Hand-selected synthetic counterexamples; not production frequency or model accuracy.",
               "python": platform.python_version(), "run_utc": datetime.now(timezone.utc).isoformat()}
    with (out / "cases.csv").open("w", newline="") as handle:
        writer = csv.DictWriter(handle, fieldnames=list(rows[0])); writer.writeheader(); writer.writerows(rows)
    (out / "summary.json").write_text(json.dumps(summary, indent=2))
    (out / "fixtures.json").write_text(json.dumps(CASES, indent=2))
    if plot:
        import matplotlib
        matplotlib.use("Agg")
        import matplotlib.pyplot as plt
        fig, ax = plt.subplots(figsize=(10, 5.6), dpi=160)
        labels, values = zip(*summary["false_acceptances"].items())
        bars = ax.barh(labels, values, color=["#2563a6", "#c16b22"], height=.5)
        for bar, value in zip(bars, values):
            ax.text(value + .12, bar.get_y() + bar.get_height()/2, f"{value} of {len(invalid)}", va="center", fontsize=13)
        ax.set_xlim(0, len(invalid) + 1); ax.set_xticks(range(len(invalid) + 1))
        ax.set_xlabel("Known-invalid fixtures accepted by the rule", fontsize=12)
        ax.set_title("Success status and output shape missed task failures", loc="left", fontsize=15, pad=22)
        ax.spines[["top", "right", "left"]].set_visible(False)
        ax.tick_params(axis="y", length=0, labelsize=12)
        ax.grid(axis="x", alpha=.18); ax.set_axisbelow(True)
        fig.text(.02, .02, "Controlled synthetic lab: 12 fixtures = 2 valid, 9 invalid, 1 unknown.\nHand-selected counterexamples; no LLM calls, customer data or production-rate estimate.", fontsize=10, color="#374151")
        fig.tight_layout(rect=(0, .13, 1, 1))
        fig.savefig(out / "false-acceptances.png", facecolor="white")
        fig.savefig(out / "false-acceptances.svg", facecolor="white")
        plt.close(fig)
    print(json.dumps(summary, indent=2))


if __name__ == "__main__":
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--out", type=Path, default=Path("results"))
    parser.add_argument("--plot", action="store_true")
    args = parser.parse_args()
    run(args.out, args.plot)

Run it in a clean directory:

python3 acceptance_lab.py --out results

The run writes three inspectable artifacts: fixtures.json contains every input and expected label, cases.csv contains each rule's decisions, and summary.json records the counts, Python version and execution timestamp. The local run used for this article completed on October 29, 2026, with Python 3.14.2. There are no random seeds, model versions or remote services involved.

To reproduce the figure in an isolated environment:

python3 -m venv .venv
.venv/bin/python -m pip install matplotlib
.venv/bin/python acceptance_lab.py --out results --plot

This also writes PNG and SVG versions of the figure. The recorded plotting environment used Matplotlib 3.11.2. The acceptance counts do not depend on that plotting library. A different plotting version can change rendering without changing the fixture decisions.

Read the decisions before reading the chart

Figure: false acceptances among the nine known-invalid, hand-selected synthetic fixtures. HTTP 2xx accepts eight; the booking-shape rule accepts seven. This is a counterexample demonstration, not an estimate of customer traffic, production reliability or model accuracy. The one unknown outcome is excluded from this denominator.

The wrong-day case illustrates an important boundary: the returned object passes the shape rule because every required field has an allowed type. Only a comparison with the request detects the date mismatch. Missing confirmation illustrates another boundary: a correct-looking final state cannot retroactively establish authorization. The duplicate fixture shows why checking that a matching record exists is weaker than checking that exactly one exists for this operation.

The cancelled and queued fixtures deliberately pass a permissive shape rule. Tightening the schema to require status == "booked" would catch those two cases. That is a useful improvement; it would not catch a wrong day, wrong customer, duplicate or fabricated receipt. You can change the rule and rerun the table to see which defects remain.

The task rule's agreement with the expected column is unsurprising: both were written for this controlled fixture set. The assertion catches an accidental mismatch while editing the lab. It does not validate the oracle independently or establish generalization to unseen tasks. Keep that distinction visible when reporting evaluation results.

Replace simulated state with evidence from one staging workflow

The first production adaptation should be narrow. Choose one operation whose desired result you can verify, such as creating a test appointment. Keep the user goal, permitted actions and observation deadline explicit. Avoid starting with a universal score for every conversation.

Record the requested date and customer reference before execution, together with the authorization or confirmation required by that workflow. After the action, use a documented read path to establish the resulting state. An echoed response is not an independent observation: if the agent and evaluator both trust the same invented receipt, they can agree about a booking that never existed.

For asynchronous work, use the provider's documented job state and terminal outcomes. Decide how long the test waits and what happens if observation remains incomplete. A pending operation should not become a completed task simply because a polling deadline elapsed. Preserve an unresolved category and reconcile later when the system exposes enough evidence.

Adapt the duplicate check to the scope of the operation. A customer can legitimately have several appointments. A real test should look for unintended additional records caused by this action, using an operation identifier, a before/after observation or another supported reconciliation method. The fixture's single-record list represents that isolated operation; it is not a rule that each customer may own only one booking.

The toy reply labels also need replacement. Here, confirmed, unavailable, error and unknown are controlled classifications. A real assistant writes natural language. For an initial integration, prefer explicit structured completion fields backed by state evidence. If a model grades the wording, calibrate it against independently reviewed examples, including ambiguous cases and plausible but false completion claims. Do not reuse a model's explanation as the sole evidence that its own action succeeded.

Preserve enough context to debug a disagreement

When the status check passes and the task check fails, developers need a path from that decision back to the relevant operation. OpenTelemetry's trace model describes linked spans for units of work. In an instrumented workflow, the useful connection is between the evaluated run and the spans or operation identifiers that explain its actions.

A compact evaluation record can keep these fields:

{
  "run_id": "synthetic-run-003",
  "contract_version": "booking-v1",
  "fixture": "wrong_day",
  "transport_check": "PASS",
  "shape_check": "PASS",
  "task_check": "FAIL",
  "reason": "requested_date_does_not_match_established_state",
  "evidence_kind": "synthetic_fixture"
}

In a real system, replace the fixture reference with appropriate evidence references and record when they were observed. Keep authorization evidence, state evidence and explanatory text distinguishable. They answer different questions. Apply the system's existing access and retention controls rather than copying full customer payloads into every evaluation record.

Use counterexamples as a release tool

This lab is useful as a small regression suite, not as a production success dashboard. Start with the known counterexamples, then add failures found in staging or support investigations after removing private data. Each addition should explain which acceptance rule it challenges and what observation establishes the expected result.

For a proposed evaluator change, compare decisions case by case. A new rule that rejects more outputs may reduce false acceptance while also rejecting valid unavailable-slot responses. Report both effects. Keep unknown outcomes separate so an apparent improvement cannot be manufactured by moving difficult cases out of the measured set without explanation.

If you later measure production reliability, define the sampling frame, time window, unit of analysis and handling of retries before presenting a rate. The nine invalid examples here are deliberately selected; dividing their counts does not supply those missing ingredients.

The immediate engineering change is concrete: add a task-specific acceptance check beside the HTTP and output checks for one workflow. Include a valid limitation response, a wrong target, a duplicate and an unobservable outcome. Run those cases before a release. A successful request is useful evidence, but completing the user's authorized task requires evidence at the task boundary too.

About the author: Shashank Agarwal is the founder and CEO of Noveum.ai and the creator of API.market. This article uses a standalone synthetic lab; it makes no claim about either company's customer results or revenue.