48 小時實測筆記：串流未送出 [DONE]，客戶端卻判定成功

Why does a truncated model answer look exactly like a finished one when the only thing you check is the HTTP status code?

That question cost me most of a weekend. I had wired a small summarizer to a streaming endpoint, watched it print a paragraph, and shipped it. Two days of field notes later, I had learned that "HTTP 200" tells you nothing about whether the answer actually completed, and that my parser was happily reporting success on streams that never sent a terminator.

Here is what I tried, what broke, and the probe harness I now run before trusting any streaming endpoint.

The symptom: a paragraph that stopped mid-thought

In my run, the response body ended cleanly at the socket level, the status line said 200, and my client wrote whatever it had buffered. Nothing raised. The output simply stopped mid-sentence, which readers interpret as a model being terse rather than a transport being over. Because I was printing tokens as they arrived, I never compared what I received against what the protocol promised me.

That is the trap with streaming: the happy path and the failure path share the same first N bytes. If you only assert on status codes and on the existence of output, both paths pass.

What I instrumented instead of what I assumed

I stopped eyeballing output and wrote a probe that records protocol facts rather than vibes. It reads the response line by line, assembles SSE frames, and records four things per request: time to first byte, whether a finish_reason appeared in any JSON payload, whether the [DONE] sentinel arrived, and how the socket ended.

# stream_probe.py — stdlib only, so it runs anywhere
import json, sys, time, urllib.request


def parse_frame(lines):
    """SSE frames are newline-delimited; data may span several lines."""
    payload = []
    for line in lines:
        if line.startswith(":"):
            continue                      # comment / keep-alive
        if line.startswith("data:"):
            payload.append(line[5:].lstrip(" "))
    if not payload:
        return None, False
    joined = "\n".join(payload).strip()
    if joined == "[DONE]":
        return None, True
    try:
        return json.loads(joined), False
    except json.JSONDecodeError:
        return {"_unparsed": joined}, False


def find_finish_reason(obj):
    if isinstance(obj, dict):
        if obj.get("finish_reason"):
            return obj["finish_reason"]
        for v in obj.values():
            hit = find_finish_reason(v)
            if hit:
                return hit
    elif isinstance(obj, list):
        for v in obj:
            hit = find_finish_reason(v)
            if hit:
                return hit
    return None


def probe(url, payload, headers, read_timeout):
    rec = {"ttfb_s": None, "frames": 0, "finish_reason": None,
           "done_sentinel": False, "unparsed_frames": 0, "error": None}
    req = urllib.request.Request(
        url, data=json.dumps(payload).encode(), headers=headers, method="POST")
    started = time.monotonic()
    frame_lines = []
    try:
        with urllib.request.urlopen(req, timeout=read_timeout) as resp:
            for raw in resp:
                if rec["ttfb_s"] is None:
                    rec["ttfb_s"] = round(time.monotonic() - started, 3)
                line = raw.decode("utf-8", "replace").rstrip("\r\n")
                if line:
                    frame_lines.append(line)
                    continue
                obj, done = parse_frame(frame_lines)
                frame_lines = []
                if done:
                    rec["done_sentinel"] = True
                if obj is not None:
                    rec["frames"] += 1
                    if "_unparsed" in obj:
                        rec["unparsed_frames"] += 1
                    else:
                        rec["finish_reason"] = rec["finish_reason"] or find_finish_reason(obj)
    except Exception as exc:              # read timeouts land here on purpose
        rec["error"] = f"{type(exc).__name__}: {exc}"
    rec["elapsed_s"] = round(time.monotonic() - started, 3)
    return rec


if __name__ == "__main__":
    url = sys.argv[1]
    body = json.loads(sys.argv[2])
    hdrs = json.loads(sys.argv[3])
    print(json.dumps(probe(url, body, hdrs, read_timeout=20.0), indent=2))

Then a soak loop, because one clean request proves almost nothing:

for i in $(seq 1 20); do
  python stream_probe.py "$URL" "$BODY" "$HEADERS" | jq -c '{ttfb_s,frames,finish_reason,done_sentinel,unparsed_frames,error}'
done | tee soak.jsonl
jq -s 'group_by(.finish_reason) | map({reason: .[0].finish_reason, n: length})' soak.jsonl

The four endings the harness separates

Once I classified results instead of staring at them, the ambiguity collapsed into a small set of observable outcomes:

1. Clean finish. A finish_reason arrived and [DONE] followed it, so the answer is complete and my parser can trust the buffer.
2. Reason without sentinel. The payload carried a finish_reason but the stream closed before [DONE], which suggests a proxy that buffers or cuts the tail frame.
3. Sentinel without reason. [DONE] arrived early, which usually means the server truncated generation server-side and reported nothing useful about why.
4. Neither. The socket closed with a partial sentence and no metadata, which is the case that burned me, and the only honest response is to mark the answer as incomplete and retry or surface it to the caller.

Only case one deserves a green check. Everything else should set an incomplete flag on the record you store, because a truncated summary that silently enters your database is worse than a visible error.

Where a free server and free models actually fit

I run this probe from a throwaway box precisely because I want the transport to be the boring part. MonkeyCode offers free model access and a free server option, which makes it a reasonable place to park a short soak loop like the one above while you are still deciding whether an endpoint deserves production traffic.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The important discipline is not to treat "free" as "verified". Whatever endpoint you point the probe at, read that provider's current limits and terms yourself on the day you run it, because quotas, model availability, and server lifetimes change and any number I copied here would be stale by the time you read it.

A decision table I now keep in the repo

What I would repeat and what I would skip

I would repeat the soak loop and the classification, because those two together turned a spooky intermittent bug into a checklist. I would skip the hours I spent reading model output for clues about whether the generation was "good", since the transport was broken the whole time and no amount of reading prose would have revealed that.

I would also repeat running the probe from a machine I do not care about. Wrapping the read in a hard timeout and letting exceptions land in the record, rather than crashing the loop, is what made the soak results readable.

Limitations and who should not bother

This workflow is for streaming text responses over SSE; if your SDK already exposes a completion flag you trust, a hand-rolled probe is mostly redundant. It also cannot tell you why a provider cut a stream, only that the frames disagree with each other, so pair it with the provider's own logs when you escalate. And if your workload is batch and non-streaming, a single content-length check will beat all of this.

If you keep one habit from these field notes, make it this: assert on the terminator, not on the status code. A stream that never says goodbye is not a finished answer, no matter how confident the first paragraph looks.