AI 串流生成的取消應是終止狀態，而非例外

A support ticket landed on a Tuesday: a user closed the tab forty seconds into a long summary, reopened the app, and found two generations billed against her workspace. Nothing in the logs was red. The job row said running, the ledger said charged, and the provider dashboard said the call had finished normally — because from the provider's point of view, nobody ever hung up.

That ticket is the whole article. We keep treating "cancel" as a piece of UI candy: a stop button that flips local state and lets the expensive part keep going. I think that is backwards. Cancellation is a distributed transaction, and it only counts when it reaches every party that can spend your money — the model provider, your persistence layer, and whoever reads the ledger afterward.

The handoff where it always breaks first

Trace a single cancel through a normal full-stack AI feature and you will find the same seam. The browser aborts the fetch, your API sees the socket drop, and then your handler does the one thing that guarantees a leak: it raises, unwinds, and never writes a terminal row. The upstream httpx call keeps reading tokens into a buffer nobody will ever consume.

Why does it survive code review? Because it looks like an exception path, and exceptions feel handled. If your generation state machine has no cancelled state, the only honest thing your code can do with a cancel is throw — and throwing is exactly what skips your finally bookkeeping in the sloppy version everyone writes first.

So the fix is not defensive try/except everywhere. The fix is to declare the state machine, then make the code obey it.

When cancelled is a first-class state, the ledger question answers itself: you spent tokens up to the moment you hung up, and you say so.

The server half: cut the socket, not the loop

Here is the shape I ship, written against FastAPI and httpx. Treat it as a skeleton: the timeouts are placeholders you should measure, and the ledger functions are your own repository calls.

# app/api/generate.py - skeleton, not a drop-in file
import httpx
from fastapi import APIRouter, Request

router = APIRouter()

@router.post("/generate/{gen_id}")
async def generate(gen_id: str, payload: Prompt, request: Request):
    await ledger.open(gen_id, prompt=payload.text)  # idempotent on gen_id
    client = httpx.AsyncClient(timeout=httpx.Timeout(60.0, read=15.0))
    final = "failed"
    try:
        async with client.stream("POST", PROVIDER_URL, json=payload.body()) as upstream:
            async for chunk in upstream.aiter_text():
                if await request.is_disconnected():
                    await client.aclose()  # hang up on the provider
                    final = "cancelled"
                    break
                await sink.write(gen_id, chunk)
            else:
                final = "completed"
    except httpx.ReadTimeout:
        final = "failed"
    finally:
        await client.aclose()
        await ledger.close(gen_id, state=final, spent=usage_for(gen_id))

The one line people skip is await client.aclose() inside the loop. Breaking out of aiter_text closes the stream in some client versions and not others, and "in some versions" is how you get a monthly bill you cannot explain. Closing twice is harmless; not closing is not.

Two honest caveats on this snippet. Polling request.is_disconnected() per chunk is cheap but not free, so at high chunk rates you want an asyncio.Event set by the disconnect watcher and an anyio race instead. And if you sit behind a proxy that buffers responses, the disconnect may reach the proxy before it reaches you, which means your handler keeps a zombie stream alive. The mitigation is a hard per-generation deadline, not hope.

The client half: tab closes are not polite

Aborting a fetch is easy. Making the abort arrive is the hard part, because a closing tab does not always flush a normal request.

const ctrl = new AbortController();
const timer = setTimeout(() => ctrl.abort(), 45_000);

try {
  const res = await fetch(`/generate/${genId}`, {
    method: "POST",
    body: JSON.stringify(payload),
    signal: ctrl.signal,
  });
  await readStream(res.body);
} catch (err) {
  if (err.name === "AbortError") await reportCancel(genId, "user_stopped");
  throw err;
} finally {
  clearTimeout(timer);
}

window.addEventListener("pagehide", () => {
  navigator.sendBeacon(`/generate/${genId}/cancel`, JSON.stringify({ reason: "tab_closed" }));
});

The explicit cancel endpoint matters more than the AbortController. A dropped socket is a hint; a POST to a cancel route is a fact you can act on idempotently, and sendBeacon fires on pagehide when a normal fetch would be cancelled by the browser itself. Your cancel route should be safe to call twice, five times, or after completion, and it should return 409 or a no-op 200 depending on your taste — but it must never create a second generation.

The drill that proves it

Here is the part I care about most, because a cancel path you have not killed is fiction. Run this as a test, not as a manual click-through.

# tests/test_cancel_path.py - one failure mode, end to end
@pytest.mark.asyncio
async def test_stream_killed_midflight_lands_as_cancelled(app, mock_provider, ledger):
    mock_provider.emit_tokens(count=50, delay=0.05)  # ~2.5s of fake stream

    async with httpx.AsyncClient(app=app, base_url="http://test") as client:
        task = asyncio.create_task(client.post("/generate/gen_1", json={"text": "hi"}))
        await asyncio.sleep(0.4)
        task.cancel()  # the client walks away
        with pytest.raises(asyncio.CancelledError):
            await task

    row = await wait_for_terminal(ledger, "gen_1", timeout=2.0)
    assert row.state == "cancelled"
    assert row.spent_tokens < 50
    assert mock_provider.last_request_aborted is True
    assert mock_provider.open_streams == 0

That last assertion is the one that catches real bugs. state == cancelled proves your bookkeeping heard you; open_streams == 0 proves you actually hung up. I have seen the first pass and the second fail, which is the worst kind of green build: clean audit trail, leaking socket.

Add a second test that replays the cancel endpoint three times after completion and asserts no duplicate ledger row. Then add a chaos case where the provider ignores your abort — a mock that keeps emitting after aclose() — and confirm your sink stops writing anyway, because you cannot control a vendor, only your own side.

Where free capacity actually helps

A drill like this needs traffic, and traffic is what free tiers are for. I run the kill-the-stream suite against MonkeyCode's free model access — the operator advertises a ten-million-token allowance — because burning a few hundred aborted completions on a paid key to prove a finally block is an absurd way to spend a Tuesday. The free server option matters for a subtler reason: running the same test against a deployed instance instead of localhost puts a real network hop between client and handler, and that hop is where disconnect detection gets honest about itself.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you want to try the drill, the open-source project is a reasonable place to start, and I would rather you bring the test than the tutorial. What I will not claim is that any free allowance makes poor cancellation hygiene acceptable — the allowance is finite and your users are not.

Who should skip this

Not every generation needs a cancel path, and pretending otherwise adds states you will never exercise. Batch jobs where nobody is watching, internal tools with three users, and anything wrapped in a queue worker that owns its own lifecycle can get away with a deadline and a dead-letter state instead of a live cancel protocol.

You should also skip the interactive cancel state machine if your provider offers no way to signal an abort and your client library cannot close the underlying socket — in that case your only real control is a server-side deadline, and you should say so in the UI rather than shipping a stop button that stops nothing.

On the other end, if you are shipping anything where a user can watch tokens arrive, the drill above is table stakes, and it is cheaper than the refund email.

Which handoff in your stack is least stable — the browser abort, the server's disconnect detection, or the provider's stream close? Give me the concrete failure state or response code you see, and I will tell you which assertion to add first.

If this was useful, the one thing I would ask: run the kill-the-stream test against your own endpoint before you deploy the next change to it, and paste the response body you get back when a stream dies mid-flight.