如何拆分 AI 延遲，區分系統路徑耗時與模型生成耗時

Wall clock is the most popular wrong number in AI tooling. You send a prompt, you stopwatch the answer, you blame the model. But three things live inside that number — queueing, the wire, and generation — and the model owns exactly one of them.

Which means a lot of "this model is slow" verdicts are really "this path is slow" verdicts. So why do we keep publishing the first as if we'd proven the second?

Here's the move I keep coming back to: record two timestamps, then report a ratio instead of a duration. A ratio survives laptops, months, and model swaps. A raw millisecond count does not.

The split

Time to first token tells you how long you waited for the system. Time from first token to last token tells you how fast the model actually wrote. Everything before the first token — DNS, TLS, auth, scheduling, batching — is what I call path tax. Everything after it is generation.

Add them and you get the number that ends up in your README, your ticket, your angry post. Only one of the two is the model's fault. Timing a model by total wall clock is like timing a phone call by how long your finger was on the button.

The harness

One file, no framework, about forty lines. Streaming stays on, because non-streaming hides the split completely — you get one blurry total and nothing to argue with.

# probe.py — run: TAG=laptop python probe.py --runs 30 --out ledger.jsonl
import argparse, hashlib, json, os, time
import requests

URL   = os.environ["CHAT_URL"]     # your endpoint, not mine
MODEL = os.environ["MODEL_ID"]
KEY   = os.environ["API_KEY"]
TAG   = os.environ.get("TAG", "local")

PROMPT = "Write a Python function that dedupes a list while preserving order."

def one_run(i, prompt=PROMPT, max_tokens=256):
    body = {"model": MODEL, "stream": True, "max_tokens": max_tokens,
            "messages": [{"role": "user", "content": prompt}]}
    t0  = time.monotonic_ns()
    r   = requests.post(URL, json=body, stream=True, timeout=120,
                        headers={"Authorization": f"Bearer {KEY}"})
    first = None
    for line in r.iter_lines(decode_unicode=True):
        if not line or not line.startswith("data:"):
            continue
        if line[5:].strip() == "[DONE]":
            break
        if first is None:
            first = time.monotonic_ns()
    t_end = time.monotonic_ns()
    if first is None:
        return {"run": i, "ok": False, "status": r.status_code}
    total_ms = (t_end - t0) / 1e6
    gen_ms   = (t_end - first) / 1e6
    return {"run": i, "ok": True, "tag": TAG, "model": MODEL,
            "ttft_ms": round((first - t0) / 1e6, 1),
            "gen_ms": round(gen_ms, 1), "total_ms": round(total_ms, 1),
            "path_tax": round(1 - gen_ms / total_ms, 3),
            "prompt_sha": hashlib.sha256(prompt.encode()).hexdigest()[:12],
            "max_tokens": max_tokens, "status": r.status_code}

if __name__ == "__main__":
    ap = argparse.ArgumentParser()
    ap.add_argument("--runs", type=int, default=30)
    ap.add_argument("--out", default="ledger.jsonl")
    a = ap.parse_args()
    with open(a.out, "a") as f:
        for i in range(a.runs):
            row = one_run(i)
            f.write(json.dumps(row) + "\n")
            print(row)

Two details matter more than they look. time.monotonic_ns() instead of time.time(), because a clock correction mid-run will quietly poison your p90. And the ledger stores a hash of the prompt rather than the prompt itself, so a rerun next month is comparable without dragging your test data around.

Notice what's missing: token counts. If your stream reports usage, capture it. If it doesn't, estimate locally and label it an estimate in the ledger. An honest estimate beats a confident blank, and a blank is what most teams actually have.

Report the tax, not the speed

# report.py — run: python report.py ledger.jsonl
import json, sys

rows = [json.loads(l) for l in open(sys.argv[1])]
rows = [r for r in rows if r.get("ok")]

def q(xs, p):
    xs = sorted(xs)
    return xs[min(len(xs) - 1, int(round(p * (len(xs) - 1))))]

tax = [r["path_tax"] for r in rows]
ttft = [r["ttft_ms"] for r in rows]
tot  = [r["total_ms"] for r in rows]
print(f"n={len(rows)}")
print(f"ttft  p50={q(ttft,.5):7.0f}ms  p90={q(ttft,.9):7.0f}ms")
print(f"total p50={q(tot ,.5):7.0f}ms  p90={q(tot ,.9):7.0f}ms")
print(f"path tax p50={q(tax,.5)*100:5.1f}%  p90={q(tax,.9)*100:5.1f}%")

path tax = 1 - gen_ms / total_ms. One number, and it tells you what you were actually measuring. If it's small, your model comparison is real. If it's large, you spent the afternoon benchmarking a load balancer.

Keep max_tokens and the prompt byte-identical across arms, run at least thirty times per arm, and record the very first run but analyze it separately — cold starts deserve their own row, not a smudge on your median.

Where the free server changes the experiment

This is the part that made me build the rig properly. MonkeyCode offers free model access and a free server option, which means you can run the same probe from two places instead of one. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I haven't handed you my numbers for it, and I won't — the endpoint, the time of day, and the load are yours, not mine. What I'm handing you is the thing that produces them.

So: TAG=laptop and TAG=freesrv, same prompt, same max_tokens, thirty runs each, then diff the path-tax distribution. If the tax collapses, the win came from the vantage point, not from the model — and knowing that is worth more than any leaderboard. The advertised allowance is ten million tokens, which is plenty of room for a run this size; treat it as a promotional claim to re-check later, not as a line in your budget spreadsheet.

What this rig cannot tell you

The ledger measures one prompt shape. Short prompt plus long output is a different experiment from long prompt plus short output, and tool-calling workloads with fat JSON payloads are a third one. Ten runs of one prompt is an anecdote wearing a lab coat.

You also can't compare across days without pinning the model identifier and logging it — models get quietly rerouted, and a silent version bump will read as a performance regression. If your product needs a hard latency SLO, or your compliance rules demand provenance you can prove, this workflow is a starting point rather than a home; go buy the guarantee. And if you're hoping a benchmark will settle an argument, it won't. It'll only tell you which of the three clocks you were arguing about.

Run it from both places and you'll learn something about the system — or something about your assumptions. Both are useful. If you do run it, drop your path-tax number in the comments; I'd rather compare distributions than adjectives.