別再怪模型：免費方案下智能體執行失敗分類探測工具

Stop Blaming the Model: A Failure-Taxonomy Probe for Agent Runs on Free Tiers

Your agent died at 2 a.m. Was the model bad? Probably not.

Most failed runs I see are misattributed. The model gets blamed for transport, quota, or a bad contract check.

Here is a short myth-busting FAQ. It ends with one probe that sorts failures into five layers.

Myth 1: A 200 response means the request worked

A status code describes transport only. It says nothing about usable output.

You can get 200 with finish_reason: "length". You can get 200 with a refusal. You can get 200 with JSON that parses but misses a required field.

Is that a successful unit of work? No. It is a received but rejected artifact.

So define acceptance in code. Parse it, validate it, hash it. Otherwise you are counting receipts, not results.

Myth 2: Retries are free

Retries are cheap on the network side. They are expensive on the side-effect side.

Retry a read and nothing changes. Retry a tool call that writes a row and now you have two rows.

That is not a model problem. That is a missing idempotency key.

My rule: every attempt carries a stable Idempotency-Key. Derive it from the task id, never from the attempt counter.

Myth 3: Latency numbers tell you which model to pick

Median latency answers one narrow question. How fast is a warm path?

That is rarely your user's path. Free-tier hosts often show two clusters: warm and cold.

Measure time-to-accepted-artifact instead. That number includes your retries and your validator.

Two models with identical token speed can land far apart here. One simply fails validation more often.

Myth 4: A bad run means a bad model

Attribute before you switch vendors. Ask which layer failed, not which model.

• Transport: timeout, DNS failure, connection reset.
• Quota: 429, 402, or an explicit rate-limit body.
• Contract: schema mismatch, truncation, refusal, empty content.
• Tool: your function raised, or returned junk.
• Task: the prompt asked for something impossible.

Only the last two are usually your prompt's fault. The contract layer is often your own validator.

The probe: one script, five layers

This script uses the Python standard library only. It records enough to classify every attempt.

I have not run it against your endpoint. Treat any output as illustrative, not as a benchmark.

# probe.py - failure-taxonomy probe (illustrative, unexecuted here)
import json, os, time, urllib.request, urllib.error, hashlib

BASE   = os.environ["BASE_URL"]     # OpenAI-compatible base, e.g. https://host/v1
MODEL  = os.environ["MODEL"]
KEY    = os.environ.get("API_KEY", "")
RUNS   = int(os.environ.get("RUNS", "8"))
PROMPT = os.environ.get("PROMPT", 'Return JSON: {"ok": true}')

def call(idem):
    body = json.dumps({
        "model": MODEL,
        "messages": [{"role": "user", "content": PROMPT}],
        "temperature": 0,
        "max_tokens": 256,
    }).encode()
    url = BASE.rstrip("/") + "/chat/completions"
    req = urllib.request.Request(url, data=body, method="POST")
    req.add_header("Content-Type", "application/json")
    req.add_header("Idempotency-Key", idem)
    if KEY:
        req.add_header("Authorization", "Bearer " + KEY)
    t0 = time.monotonic()
    try:
        with urllib.request.urlopen(req, timeout=120) as r:
            return r.status, r.read(), (time.monotonic() - t0) * 1000, ""
    except urllib.error.HTTPError as e:
        return e.code, e.read(), (time.monotonic() - t0) * 1000, "http_error"
    except Exception as e:
        return None, b"", (time.monotonic() - t0) * 1000, type(e).__name__

def classify(status, schema_ok):
    if status is None:           return "transport"
    if status == 429:            return "quota"
    if status in (401, 403):     return "auth"
    if status and status >= 500: return "upstream"
    if status != 200:            return "contract"
    if not schema_ok:            return "contract"
    return "ok"

task_id = os.environ.get("TASK_ID", "probe-1")
ledger = []

for attempt in range(1, RUNS + 1):
    idem = task_id               # stable across retries, on purpose
    status, raw, ms, err = call(idem)
    schema_ok, reason = False, ""
    try:
        payload = json.loads(raw.decode() or "{}")
        choice = payload["choices"][0]
        reason = choice.get("finish_reason", "")
        json.loads(choice["message"]["content"])   # your real contract check goes here
        schema_ok = True
    except Exception as e:
        reason = reason or type(e).__name__
    ledger.append({
        "attempt": attempt, "idem": idem, "status": status,
        "ms": round(ms), "bytes": len(raw), "finish": reason,
        "err": err,
        "sha256": hashlib.sha256(raw).hexdigest()[:12],
        "layer": classify(status, schema_ok),
    })

for row in ledger:
    print(json.dumps(row))

print("--- layer histogram ---")
for layer in sorted({r["layer"] for r in ledger}):
    print(layer, sum(1 for r in ledger if r["layer"] == layer))

Run it with three environment variables and a repeat count:

BASE_URL=https://your-endpoint/v1 MODEL=your-model RUNS=12 python3 probe.py

The ledger is the whole point. One row equals one attempt with one label.

How to read the ledger

• Many contract rows with finish: "length"? Raise the output budget, or shrink the ask.
• transport rows at fixed intervals? Suspect an idle timeout, not the model.
• quota mixed with ok? You need pacing, not a new vendor.
• Identical sha256 values at temperature: 0? Cache them, do not re-run.
• Zero ok rows and clean transport? Your contract check is probably wrong.

Does reading the labels change your conclusion? It usually changes mine.

A decision table you can argue with

Keep this table next to the script. It ends the "which model is best" debate quickly.

The debugging workflow, in order

1. Reproduce with RUNS=1 and a fixed prompt.
2. Read the response body, not just the status.
3. Log finish_reason and output byte count.
4. Replace my contract check with your real one.
5. Re-run with RUNS=12 and read the histogram.
6. Fix only the top layer. Then measure again.
7. Keep the ledger as a regression fixture.

Steps five and six get skipped all the time. That is where the time is saved.

Where a free tier actually fits

I would run this probe before trusting any free endpoint with a batch job.

That is the useful part of free model access plus a free server option. You get a place to run the probe before provisioning anything.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode lists free model access and a free server option. Those two claims are operator-supplied, and I am not extending them.

I will not guess at model names, quota, hardware, or duration. Measure those with the script above, then decide.

Use the free tier for the probe and the small jobs. Keep the layer table honest either way.

Limitations, and who should skip this

• The script assumes an OpenAI-compatible chat endpoint. Yours may not be.
• It does not test concurrency. Add a parallel runner if that is your risk.
• It cannot prove a tier stays free. Re-run it on a schedule instead.
• It says nothing about output quality. That needs a separate eval.
• The Idempotency-Key header only helps if your server honors it.

Skip this if you already trace each attempt with layer labels. Skip it for a single one-off call.

Also skip it if you need throughput guarantees today. A probe measures, it does not promise.

The question I would ask first

Next time an agent run fails, ask: which layer failed? Not: which model should I try?

Write the layer into your logs and your error messages. The argument gets shorter every time.