AI Token 串流一次到齊：如何排查緩衝並修復

Support tickets about AI features rarely say "your relay buffers the response." They say something like this: the answer is correct, it just shows up in one lump after a long pause, and my screen reader reads the whole paragraph in a single breath.

Those look like two complaints — one about latency, one about accessibility — but in the failures I have traced, they share one root cause: a stream that was correct on the wire arrived at the client in a single chunk. The model was never slow. The bytes were simply held somewhere in the middle and released together.

That failure shape is common enough that it deserves a checklist rather than a hunch. So here is the debugging path I would hand to a teammate, with a reduced reproduction you can run, the places buffering actually hides, and the accessibility regression that bites even after the latency looks fixed.

The scenario below is a composite of that failure shape rather than one specific incident; the reproduction and commands are real artifacts you can run yourself.

Write the state table before you touch the network tab

Streaming UI has more states than people admit, and buffering breaks exactly one of them. If you cannot name the states, you will "fix" the wrong one.

The bug in question shows up as connecting that appears to last 30 seconds and then a jump straight to complete, skipping visible streaming entirely. Modeling it as a typed state machine makes the symptom precise:

type StreamState =
  | { phase: 'idle' }
  | { phase: 'connecting'; startedAt: number }
  | { phase: 'streaming'; firstTokenAt: number; chars: number }
  | { phase: 'complete'; durationMs: number }
  | { phase: 'cancelled'; reason: 'user' | 'timeout' }
  | { phase: 'error'; retryable: boolean; httpStatus?: number };

If streaming never renders, the question becomes narrow and testable: did the client receive frames progressively, or did it receive one frame at the end?

Rule out the model before you blame the model

A slow provider and a buffered relay produce the same user complaint. Separate them with timestamps, not with opinions.

1. Hit the upstream model endpoint directly and timestamp every line as it arrives.
2. Hit your own relay endpoint the same way.
3. Compare the gaps between lines, not the total duration.
4. Repeat step 2 through a deployed host, not through localhost.

# Timestamp each line as it lands. If every timestamp is identical,
# you are seeing one buffered chunk, not a slow stream.
curl -N -H 'Accept: text/event-stream' https://your-relay.example/stream \
  | while IFS= read -r line; do printf '%s %s\n' "$(date +%s.%3N)" "$line"; done

# Inspect the response headers for a smoking gun.
curl -sS -D - -o /dev/null -H 'Accept: text/event-stream' https://your-relay.example/stream

Two details are worth memorizing. First, curl -N disables curl's own output buffering; without it you can manufacture the bug yourself. Second, a content-encoding: gzip header on an event stream is a strong hint that a compression layer is sitting between the model and the browser.

The three places a stream actually gets held

In my experience the bytes stop moving in one of three places, and they are easy to test in order:

• Compression middleware. A generic gzip layer waits for a threshold before flushing, which is exactly wrong for latency-critical frames.
• A reverse proxy. nginx's proxy_buffering defaults to on, so the proxy happily accumulates your stream until the upstream response ends.
• Client-side code. A stray await response.json(), an intermediate API route that buffers the whole body, or a service worker that clones the response will all destroy progressive rendering.

Note what is not on the list: the browser's fetch reader. In the failures I have triaged, response.body.getReader() was doing its job and handing over whatever the network delivered — which was one chunk.

A relay that lies, in about thirty lines

The fastest way to internalize this bug is to build it on purpose. This relay emits twenty frames 300 ms apart and wraps them in gzip, so the client sees almost nothing until the end:

// server.mjs — deliberately buffered reproduction
import express from 'express';
import compression from 'compression';

const app = express();
app.use(compression()); // <-- the bug: frames accumulate instead of flushing

app.get('/stream', async (req, res) => {
  res.setHeader('Content-Type', 'text/event-stream');
  res.setHeader('Cache-Control', 'no-cache, no-transform');
  res.setHeader('Connection', 'keep-alive');
  // Note: no res.flushHeaders() here on purpose.

  for (let i = 0; i < 20; i++) {
    res.write(`data: ${JSON.stringify({ i, at: Date.now() })}\n\n`);
    await new Promise((resolve) => setTimeout(resolve, 300));
  }
  res.end();
});

app.listen(3000, () => console.log('http://localhost:3000/stream'));

Run it, then timestamp the output with the curl loop above. On a fast local connection you may still see reasonable spacing — which is precisely why this class of bug survives code review and dies in production.

The fix, and why "just add flush" is only half a fix

The transport fix is short, and it is about telling every layer that latency beats compression ratio:

import compression from 'compression';
import zlib from 'node:zlib';

app.use(
  compression({
    // Flush after each write instead of waiting for the internal buffer.
    flush: zlib.constants.Z_SYNC_FLUSH,
    // Never compress event streams: they are small and latency-critical.
    filter: (req) => req.headers.accept !== 'text/event-stream',
  })
);

app.get('/stream', async (req, res) => {
  res.setHeader('Content-Type', 'text/event-stream');
  res.setHeader('Cache-Control', 'no-cache, no-transform');
  res.setHeader('X-Accel-Buffering', 'no'); // nginx: do not buffer this response
  res.flushHeaders(); // unblock the client reader immediately
  // ...write frames...
});

The half that people skip is the accessibility half. Progressive frames are not automatically a good screen reader experience: if you point aria-live="polite" at the whole growing transcript, every frame risks a new announcement, and some readers will restart mid-sentence. Put the transcript outside the live region, and announce from a separate node on a timer or sentence boundary instead.

<!-- The transcript is not a live region. -->
<div id="transcript" role="log" aria-live="off"></div>

<!-- A small, separate node handles announcements. -->
<p id="stream-status" role="status" aria-live="polite" aria-atomic="true"></p>

// Announce at most once per ~1500 ms, and always a complete sentence fragment.
let lastAnnounced = 0;
function maybeAnnounce(text) {
  const now = performance.now();
  if (now - lastAnnounced < 1500) return;
  lastAnnounced = now;
  document.getElementById('stream-status').textContent = text;
}

Why localhost cannot reproduce this, and what to do about it

Loopback has no compression proxy, no cellular handoff, and no corporate middlebox. DevTools throttling changes bandwidth, not buffering semantics — those are different variables, and only one of them triggers this bug.

So the network path has to be real. That is the part of the workflow where I used MonkeyCode (Disclosure: This article was prepared as part of MonkeyCode's product outreach.), an open-source AI coding project whose operator states that it includes free model access — described as a ten-million-token allowance — and a free server option for running small services. Both of those are operator-supplied claims rather than verified benchmarks, so check the project's current terms yourself before relying on either one.

Why those two things are the relevant ones here:

1. A real model stream gives you realistic timing for free. Faking tokens with setTimeout teaches you nothing about inter-token gaps. A genuine model endpoint produces the uneven cadence that actually stresses a buffered relay.
2. A reachable host gives you a real path. A server you can open from a phone on cellular exercises a proxy and a handoff that localhost never will.
3. A second environment gives you a control. If the bug reproduces on the hosted relay but not on loopback, the culprit is in the deployment path, not in your reader loop.

The debugging sequence I would use: run the reproduction locally, confirm the timestamps cluster; deploy the same relay to the hosted environment; open it from a phone off Wi-Fi; timestamp both with curl; then apply the flush and compression fixes and confirm the timestamps spread out.

Limitations, and who should skip this

• This workflow diagnoses delivery, not model quality. If the first token genuinely takes 20 seconds, buffering is not your problem.
• Free tiers change. Treat the availability claims above as a snapshot of a vendor description, not a guarantee, and never put real credentials or user data into a throwaway reproduction host.
• Do not ship the reproduction relay to production traffic. It has no auth, no rate limiting, and no cancellation handling.
• If your product's output is short and non-progressive by design, this whole class of bug is irrelevant to you.

The QA matrix I would run before closing the ticket

If you reproduce a variant of this, the details that matter most are your browser and OS build, your screen reader and version, and the exact transition that failed — for example, "focus left the Stop button when the stream ended on a 4G handoff." Those three facts turn a vague "streaming feels broken" report into a fix.

If you want to poke at the reproduction, the relay above runs unchanged on a small free host, which is a low-risk way to see the buffering behavior with your own eyes.