模型回應截斷後仍通過 JSON 解析，不完整結果因而未被發現

A truncated model response parsed as valid JSON. That is the entire bug.

HTTP said 200. The parser said fine. The job wrote a partial patch and moved on.

No error line. No stack trace. No alert. Just a quieter, smaller result.

I want to trace that shape properly. It is not a model quality problem. It is a completeness problem wearing a success code.

I will use a free-tier setup as the running example. Free tiers expose truncation and wall-clock limits quickly. MonkeyCode offers a free model access path and a free server option.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I am not quoting a token figure here. Quotas change, and a stale number is worse than none. Check the current limits on the project page before you size a job.

The snippets below are illustrative. Treat them as a reproduction plan, not a transcript.

Why the parser is the wrong oracle

JSON validity answers one question. Did these bytes form a tree? It says nothing about intent.

Ask the harder question. Did the model finish the thought you asked for?

A response cut at a token boundary can still close its braces. Models often end with a tidy } just before the limit hits. Your schema check passes. Your business logic proceeds.

So the first rule is simple. Never let parse success mean request success.

Step 1: Capture the bytes before anything parses them

Persist the raw body first. Parse from the saved copy, not the live stream.

Then you can replay the same bytes in a unit test. You can also diff a good run against a bad one.

curl -sS -D headers.txt \
  -o body.raw.json \
  -w 'status=%{http_code} bytes=%{size_download} time=%{time_total}\n' \
  "$ENDPOINT"

Read headers.txt before you read the JSON. Status, content length, and routing headers live there.

If bytes looks suspiciously round, treat it as a signal. Round numbers usually mean a cap, not a model decision.

Step 2: Read the finish reason, not the status

Most chat APIs return a finish or stop reason. length means the model hit a cap. stop means it chose to end.

That field is the cheapest completeness check you get. Log it next to every request id.

# illustrative

def classify(resp: dict) -> str:
    choice = resp["choices"][0]
    reason = choice.get("finish_reason")
    if reason == "length":
        return "truncated"
    if reason == "stop":
        return "complete"
    return f"unknown:{reason}"

Now wrap your call site. Raise on truncated. Do not fall through to the parser.

What if the provider does not expose a reason? Then you need a different terminator. That is step 3.

Step 3: Demand an explicit terminator token

Ask the model to end with a sentinel line. Something like END_OF_PAYLOAD on its own.

Then check for the sentinel before parsing. No sentinel, no acceptance. This works on any endpoint, free or paid.

# illustrative
import json

SENTINEL = "END_OF_PAYLOAD"

def accept(raw: str) -> dict:
    if not raw.rstrip().endswith(SENTINEL):
        raise ValueError("no terminator: treat as truncated")
    payload = raw.rstrip()[: -len(SENTINEL)].strip()
    return json.loads(payload)

Two guards now. The finish reason, and the sentinel. Belt and braces, because they fail in different ways.

Can a model hallucinate the sentinel mid-output? Occasionally, yes. That is why you also cap output length and validate required keys.

Step 4: When nothing parses, print a window, not the blob

Dumping megabytes into a terminal is slow debugging. Print the head, the tail, and the byte count.

The tail is where the evidence lives. A clean } at the end suggests a silent cap. A dangling comma suggests a hard cut.

wc -c body.raw.json
head -c 200 body.raw.json; echo
tail -c 200 body.raw.json; echo

Then compare two runs. One short prompt, one long prompt. Same endpoint, same parameters.

If only the long prompt truncates, you found a cap. If both truncate at the same byte count, you found a hard ceiling.

Step 5: Separate transport retries from model retries

These are different failures and need different logic. A 502 deserves a retry. A truncated generation usually deserves a smaller ask.

Mixing them creates the worst outcome. You retry a truncation five times and burn your allowance on identical failures.

Keep a local ledger. One line per attempt, with reason and byte count. Then you can see the pattern without guessing.

# illustrative
import json, pathlib, time

LEDGER = pathlib.Path("attempts.jsonl")

def record(attempt: int, outcome: str, nbytes: int) -> None:
    row = {"t": time.time(), "attempt": attempt,
           "outcome": outcome, "bytes": nbytes}
    with LEDGER.open("a") as fh:
        fh.write(json.dumps(row) + "\n")

Read that file after a failed run. Ten identical truncated rows mean stop retrying. Shrink the task instead.

Step 6: Checkpoint before the wall clock decides for you

Free servers often sleep, suspend, or recycle. That is normal for a zero-cost tier. Plan for it instead of fighting it.

The rule I follow is blunt. If a job takes more than one request, it must be resumable.

Write state after every completed unit. Make the unit idempotent. Then a wake-up only costs you the unit in flight.

# illustrative
import json, pathlib

def run_units(units, do_one, state_path="state.json"):
    p = pathlib.Path(state_path)
    done = set(json.loads(p.read_text())) if p.exists() else set()
    for uid in units:
        if uid in done:
            continue
        do_one(uid)
        done.add(uid)
        p.write_text(json.dumps(sorted(done)))

This is boring code. It is also the difference between a long job and a lost afternoon.

Decision table

Treat the table as a starting point. Your data rules decide more than any benchmark does.

Three limitations worth stating plainly

First, free allowances change. Nothing here should be read as a guarantee of quota, speed, or availability.

Second, a sentinel check catches truncation, not wrongness. A complete, well-formed, incorrect answer still passes.

Third, checkpoints add write amplification. For tiny jobs, the ledger is more code than the task.

Who should skip this? Teams with strict retention rules. Teams with hard p99 targets. Teams that cannot store intermediate state anywhere.

The takeaway

Success codes lie. Completeness is a property you have to measure.

Log the finish reason. Demand a terminator. Retry on transport, not on truncation. Checkpoint before the clock runs out.

If you want a zero-cost place to practice that loop, the free model path is the lowest-risk starting point. Run one long prompt and one short prompt first, then compare the byte counts.

That single comparison teaches more than any status code will.