用免費模型標註 Agent 追蹤片段，再以人工標註集把關

Most trace tooling answers what happened. Almost none answers which 40 spans of this 3,000-span run deserve a human's next hour. A cheap classifier can answer that — but only if you measure it before you trust it. The loop below builds bounded span cards, labels them with a free model endpoint, gates those labels against a hand-labeled slice, and only then lets them reorder your debug queue.

This week's AI feed is mostly capability talk. The unglamorous counterpart is verification: a labeler you have not tested is a random number generator with good manners.

Start from your own failure taxonomy

Do not borrow a label set from an article, including this one. Pull it from your last five postmortems. In my traces the recurring buckets are tool contract breaks (the tool returned a shape the prompt did not expect), prompt drift (the model stopped following a constraint mid-run), timeouts that were retried silently, and context loss after a handoff. Four buckets and an unknown escape hatch is enough to triage with.

unknown is not a failure mode. It is the honest answer when the card carries no evidence, and its rate is the first number you should look at. A labeler that answers unknown 60% of the time is telling you your cards are too thin, not that your agent is mysterious.

Card first, prompt second

A span record is usually 2–20 KB of JSON. Sending that verbatim to any model is wasteful and it invites the model to summarize rather than classify. The card is a fixed-byte projection: identity, kind, status, duration, and a truncated slice of input-to-output text.

import json

CARD_TEXT_BYTES = 320

def card(span):
    text = (span.get("input_summary") or "") + " -> " + (span.get("output_summary") or "")
    text = text.encode("utf-8")[:CARD_TEXT_BYTES].decode("utf-8", "ignore")
    return {
        "id": span["span_id"],
        "kind": span["kind"],        # "llm" | "tool" | "handoff"
        "tool": span.get("tool_name"),
        "status": span["status"],    # "ok" | "error" | "timeout"
        "dur_ms": span["dur_ms"],
        "text": text,
    }

Truncation is deliberate and it must be visible. If a span is misclassified because the evidence was cut off at byte 320, raising the byte budget should fix it — that is a falsifiable hypothesis, which is more than most prompt edits give you.

One span in, one JSON object out

The client is a callable, not an SDK import. That keeps the harness portable across whatever endpoint you point it at and keeps the retry policy in your code where you can test it.

LABELS = ["tool_contract", "prompt_drift", "timeout", "context_loss", "unknown"]

PROMPT = (
    "You label one agent span. Reply with JSON only: "
    '{"label": <one of ' + ", ".join(LABELS) + '>, "why": "<=12 words"}. '
    "Use evidence in the card only. If the card lacks evidence, answer unknown."
)

def label(llm, span, retries=2):
    c = card(span)
    msgs = [{"role": "user", "content": PROMPT + "\n\nCARD:\n" + json.dumps(c)}]
    for _ in range(retries + 1):
        try:
            raw = llm(msgs)                      # your client, one function
            obj = json.loads(raw[raw.index("{"): raw.rindex("}") + 1])
            if obj.get("label") in LABELS:
                return {"span_id": c["id"], "label": obj["label"], "why": obj.get("why", "")}
        except Exception:
            continue
    return {"span_id": c["id"], "label": "unknown", "why": "parse_fail"}

Parse failures collapse into unknown on purpose. A silent fallback that pretends to classify is worse than a bucket you can count. Run it over a sample rather than the whole trace: python -m triage.run --trace run.jsonl --sample 200 --out labels.jsonl.

The gate is the entire point

Hand-label 40–60 spans you have already debugged, once. Then measure the labeler against them, per class, not just overall.

def gate(gold, pred, floor=0.70):
    by = {}
    for sid, g in gold.items():
        p = pred.get(sid, "unknown")
        slot = by.setdefault(g, [0, 0])
        slot[0] += 1
        slot[1] += int(p == g)
    recall = {k: round(v[1] / v[0], 2) for k, v in by.items()}
    overall = round(sum(v[1] for v in by.values()) / len(gold), 2)
    return {"overall": overall, "recall": recall, "pass": overall >= floor}

The floor is a knob you calibrate, not a law. What matters is the failure direction: a labeler that misses timeout sends you chasing prompts, while one that over-fires tool_contract floods the queue with false leads. Look at the per-class recall before you look at the average, and freeze the gold set so future prompt edits are comparable. The snippets here are illustrative — I have not published numbers for them, and you should not adopt anyone's thresholds without running your own gold set.

Where the free tier actually fits

Labeling is high-volume, low-stakes, and its output is a hint rather than a verdict. That is the shape of work a free model endpoint absorbs well, and it is the reason I put labeling there instead of on the expensive path. MonkeyCode exposes free model access and a free server option; the operator describes a free token allowance in the ten-million range at the time of writing.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Quotas, model line-ups, and server terms move, so verify the current page before you wire any of it into CI, and never treat a free allowance as a capacity guarantee. Redact before you send: strip bearer tokens, customer identifiers, and file contents from input_summary and output_summary at write time, not at card time. A span that never stored the secret cannot leak it to a labeler.

The last row is where a free server earns its place: replay of a shortlisted span often needs to re-reach the same tools, and that is a job for a machine you can throw away. Keep the replayer's credentials scoped to a sandbox and log the replay's own spans separately, or you will merge two runs into one trace and lose the comparison.

Limitations, and who should skip this

If you debug fewer than a few hundred spans a week, read them. The harness costs more than it saves, and hand reading builds the intuition the taxonomy needs anyway. If your spans carry regulated data that cannot leave your network, the labeling pass has to stay local. If your tool schemas change weekly, the labeler drifts with them and the gate will fail often enough to be noise.

Agreement is not correctness. A labeler that matches your gold set has matched your judgment on that slice, and nothing more. Treat it as a queue sorter: it decides what you read first, never what you conclude.

If you want to try the loop without standing up infrastructure, the free model endpoint and free server option are the cheapest place to start — point the llm callable at the former, run the gate locally, and only move replay to the latter once the shortlist is stable.