Python requests 的 timeout=60 為何未能阻止串流 LLM 執行 38 分鐘

My nightly summarization job had timeout=60 on every request. One call ran for 38 minutes and 12 seconds anyway. Nothing crashed and nothing logged an error. The 6:00 a.m. report just didn't show up.

The Python requests timeout is not a deadline for the whole call. It caps how long you wait for the next bytes. Point it at a streaming LLM endpoint that emits a token every 70 milliseconds and that timeout never fires.

This post covers what I found over 14 nights and 33,600 calls, how I tracked it down, and the 20 lines that fixed it.

TL;DR

• requests.post(..., timeout=60) sets a connect timeout and a read timeout. The read timeout is the longest allowed gap between bytes, not the total duration.
• A streaming LLM response sends bytes constantly, so it never trips the read timeout, even when the model is stuck in a repetition loop for half an hour.
• httpx works the same way: Timeout(60) covers connect, read, write and pool separately. There is no total-time cap by default.
• Fix: set max_tokens, keep a short read timeout, and enforce a wall-clock deadline yourself (a check inside the stream loop, or asyncio.timeout() around the whole call).
• After the fix: 0 missed reports in the next 14 nights. The longest call dropped from 38m12s to 1m51s.

The setup

Nothing exotic here:

• A 7B instruct model served by vLLM through its OpenAI-compatible endpoint
• A Python batch job with 6 worker threads
• About 2,400 support tickets per night, each summarized into a short JSON blob
• Streaming on, so I could log time-to-first-token

The call looked like this, and I'd guess half the LLM scripts on GitHub look the same:

r = requests.post(URL, json=payload, stream=True, timeout=60)
for line in r.iter_lines():
    if line.startswith(b"data: ") and line != b"data: [DONE]":
        chunk = json.loads(line[6:])
        parts.append(chunk["choices"][0]["delta"].get("content") or "")

I read timeout=60 as "this call dies after a minute." That was wrong.

What does the timeout parameter in Python requests actually do?

The Python requests timeout sets two limits: how long to wait for the TCP connection, and how long to wait between bytes on the socket. Passing one number sets both to that value. Passing a tuple like (5, 30) sets them separately. Neither one limits total request time.

The requests docs say this outright. The read timeout is the number of seconds the client will wait for the server to send a byte. Every byte that arrives resets that clock.

For a normal JSON API this almost never matters. The server is silent until it has an answer, so "time between bytes" and "total time" are basically the same number.

Streaming changes that. A server streaming tokens is never silent for long. My model produced a chunk roughly every 70 ms under load, so the read timeout restarted about 14 times a second. A 60-second read timeout couldn't fire unless the model stopped completely.

Why would an LLM stream run for 38 minutes?

It ran that long because the model got stuck repeating itself, and nothing told it to stop. I hadn't set max_tokens. vLLM's OpenAI-compatible server then allows generation up to whatever context window is left after the prompt. My context was 32k tokens, the prompts were about 1.5k, and that left roughly 30,000 tokens of room.

At around 15 tokens per second per request under batch load, 30,000 tokens takes about 33 minutes. My worst call ran 38. The numbers line up.

The outputs from those calls looked like what you'd expect:

"summary": "Customer reports login failure. Customer reports login failure.
Customer reports login failure. Customer reports login failure. ...

That repeated a few thousand times. Some tickets (very long ones, pasted stack traces, logs full of near-identical lines) push small models into these loops. It's rare, but over 2,400 calls a night it happens.

How rare, and how bad?

Here's what 14 nights of logs showed:

0.18% sounds harmless until you remember there were only 6 workers. When 2 or 3 runaway calls landed on the same night, a third to half of my capacity spent half an hour generating "Customer reports login failure." The other workers also slowed down, because every runaway sequence took a slot in vLLM's batch and a chunk of KV cache.

On the 3 late nights, 4 or more workers were stuck at once.

How I found it

The job wasn't failing, so I had no errors to read. What I had:

1. py-spy dump on the stuck process. All 6 threads were sitting in iter_lines() → urllib3 → socket.recv. So they weren't deadlocked. They were busy receiving data.
2. vLLM's /metrics endpoint. vllm:num_requests_running sat at 6 with nothing waiting. The server thought it was doing useful work.
3. Logging output length. Normal summaries came out at 150 to 400 tokens. The 61 slow calls averaged about 21,000.

Then I wrote a 15-line test server that sent one SSE line every 30 seconds and pointed requests.post(..., timeout=60) at it. It sat there for an hour until I killed it. That settled it.

How do you set a total timeout for a streaming request in Python?

Set a total timeout by enforcing a wall-clock deadline in your own code. Neither requests nor httpx gives you one. Check elapsed time on every chunk, cap generation with max_tokens, and close the connection when you give up so the server stops generating.

Here's what I run now:

import json
import time
import requests

def stream_completion(payload, deadline_s=90, read_timeout=20):
    payload = {**payload, "max_tokens": 1024, "stream": True}
    start = time.monotonic()
    parts, finish = [], None

    with requests.post(URL, json=payload, stream=True,
                       timeout=(5, read_timeout)) as r:
        r.raise_for_status()
        for line in r.iter_lines():
            if time.monotonic() - start > deadline_s:
                raise TimeoutError(f"exceeded {deadline_s}s wall clock")
            if not line.startswith(b"data: ") or line == b"data: [DONE]":
                continue
            choice = json.loads(line[6:])["choices"][0]
            parts.append(choice["delta"].get("content") or "")
            finish = choice.get("finish_reason") or finish

    return "".join(parts), finish

Four things changed, and each one matters:

• max_tokens=1024. My longest legitimate summary was about 600 tokens. This alone puts a ceiling on the runaway case at roughly 70 seconds.
• timeout=(5, 20). Short connect timeout, moderate read timeout. This still catches a server that goes completely silent.
• The deadline check inside the loop. This is what actually caps total time. It only runs when a chunk arrives, so the true worst case is deadline_s + read_timeout, here 110 seconds. Know that number before you promise anyone an SLA.
• The with block. Leaving it closes the socket. vLLM notices the disconnect and aborts the request, so the GPU stops generating junk for nobody. My server logs showed the aborts.

Is there a hard cap that doesn't depend on chunks arriving?

Yes. With async code, wrap the entire call in asyncio.timeout() (Python 3.11+). It cancels the task at the deadline whether or not bytes are flowing:

import asyncio
import httpx

async def stream_completion(client: httpx.AsyncClient, payload):
    async with asyncio.timeout(90):
        async with client.stream("POST", URL, json=payload) as r:
            async for line in r.aiter_lines():
                ...

Cancellation unwinds the async with and closes the stream. This is my preferred version. I kept the sync one above because the batch job is threaded and I didn't want to rewrite it at 2 a.m.

If you use an SDK built on httpx, like the OpenAI Python client, read what its timeout argument means for streams in the version you have installed. Don't assume it's a total deadline. Wrap the call in your own deadline either way. It costs one line.

What about the 58 calls that hit max_tokens?

Capping generation doesn't fix the loop. It just makes the loop cheap. In the 14 nights after the fix, 58 calls ended with finish_reason == "length", which in this job always meant a repetition loop.

I handle those explicitly now:

• Never parse them as valid output. A truncated JSON blob full of repeated sentences is garbage.
• Retry once at a slightly higher temperature (0.3 instead of 0). 49 of 58 came back clean.
• Send the other 9 to a manual review queue with the ticket ID.

That's 9 tickets a fortnight for a person to look at, instead of 9 GPU-hours of "Customer reports login failure."

Results after the fix

Same job, same model, next 14 nights:

• Longest call: 1m 51s (it hit max_tokens, then went through the retry path)
• Calls over 5 minutes: 0
• Late reports: 0
• Total nightly runtime dropped by about 11%, mostly because runaway sequences stopped taking batch slots from healthy requests

The checklist I now apply to every LLM HTTP call

1. Always set max_tokens. A server default of "whatever context is left" is not a limit.
2. Use a tuple timeout: short connect, moderate read.
3. Add a wall-clock deadline that you enforce yourself.
4. Close the stream when you give up so the server stops generating.
5. Treat finish_reason == "length" as a failure until proven otherwise.
6. Log output token counts. That's how the 21,000-token outliers showed up.

So why didn't timeout=60 stop my request?

Because the Python requests timeout only limits connecting and the gap between received bytes. It never limits total request duration. A streaming LLM response sends a byte every few milliseconds, so the read timeout keeps resetting and a call can run as long as the model keeps generating: 38 minutes in my case, with no max_tokens set. To actually bound a streaming call, set max_tokens, keep a short read timeout, enforce a wall-clock deadline in your own loop or with asyncio.timeout(), and close the connection when the deadline hits so the server stops generating.

Written by the developer behind Preterview, an interview prep platform.