Kimi K3.1 Pro 回應無聲截斷：如何偵測並處理

You point an existing OpenAI-compatible client at Kimi K3.1 Pro and it works on the first try. Then a long generation comes back cut off mid-sentence, and nothing in your code complained. The request succeeded, the stream closed cleanly, and your user got half an answer.

By the end you can detect a truncated stream, size requests against the documented context limit, and choose between the Pro and Highspeed variants with evidence instead of vibes.

The truncation you do not see

Every chat completion comes back with a finish_reason. It is stop when the model finished naturally and length when it hit your max_tokens cap. In a non-streaming call you would see it on the response object. In a streaming call it arrives on the final chunk, and most hand-written stream loops only read delta.content.

That is the failure. Your loop consumes text, the connection closes, and finish_reason == "length" lands in a chunk you never inspected. The model had more to say and your application stored a truncated answer as if it were complete.

The max_tokens parameter makes this worse. Leave it unset and some providers apply a conservative default. Set it too high and a long prompt pushes you over the context limit, which fails the whole request instead of truncating it.

Catch finish_reason in the stream

This loop reads the text and records the termination reason from the last chunk.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.example.com/v1",
    api_key="sk-...",
)

stream = client.chat.completions.create(
    model="kimi-k3.1-pro",  # use the exact ID your provider lists
    messages=[{"role": "user", "content": user_prompt}],
    max_tokens=2048,
    stream=True,
)

finish_reason = None
for chunk in stream:
    choice = chunk.choices[0]
    if choice.delta.content:
        yield choice.delta.content
    if choice.finish_reason:
        finish_reason = choice.finish_reason

if finish_reason == "length":
    raise RuntimeError("Generation hit the token cap mid-output")

The check runs after the loop, not inside it, because the final chunk often carries finish_reason with empty content. Raising here turns a silent truncation into a visible failure your retry path can handle.

Model IDs differ between providers, so confirm the string against your provider's model list before shipping. A typo returns a 404 that looks like an auth problem.

Choose the variant on purpose

Kimi ships K3.1 Pro in two variants, and they are separate model strings rather than a speed setting on one endpoint. Swapping the string changes latency and output, so treat the choice as a real decision.

Do not assume Highspeed returns the same text faster. Run your own parity check on a sample of real prompts before you migrate anything that feeds a downstream parser.

def parity_report(prompt, model_a, model_b):
    a = client.chat.completions.create(
        model=model_a, messages=prompt, temperature=0
    ).choices[0].message.content
    b = client.chat.completions.create(
        model=model_b, messages=prompt, temperature=0
    ).choices[0].message.content
    return {"identical": a == b, "a": a, "b": b}

Pin the model string in one constant. If a variant change silently alters output, you want the diff to be a deliberate commit, not a config drift nobody noticed.

Budget tokens before you send

Truncation and request failure have opposite causes. You fail when the prompt plus max_tokens exceeds the context limit, and you truncate when max_tokens is smaller than the answer needs. Budget explicitly against the limit your provider documents.

def budgeted_max_tokens(prompt_tokens, context_limit, cap=4096):
    remaining = context_limit - prompt_tokens
    if remaining <= 0:
        raise ValueError(f"Prompt exceeds the context limit by {-remaining} tokens")
    return min(cap, remaining)

Count the prompt with the same tokenizer the provider uses, not a character heuristic. A rough estimate is fine for dashboards, but the request that crosses the line is the one you measured badly.

Retry rate limits without duplicating side effects

A retry loop that re-sends a request containing tool calls can fire a side effect twice. Rate-limit errors are safe to retry; a half-executed tool call is not.

import time

def call_with_retry(fn, attempts=4, base=1.0):
    for attempt in range(attempts):
        try:
            return fn()
        except Exception as exc:
            if attempt == attempts - 1 or "429" not in str(exc):
                raise
        time.sleep(base * 2 ** attempt)

This retries only on the rate-limit error and re-raises everything else. Give tool calls their own idempotency key so a retry is a duplicate detection problem rather than a double-charge problem.

Key takeaways

• A streaming response only reveals finish_reason on the final chunk, so check it after your loop ends.
• finish_reason == "length" means the model hit your cap, not that it finished; treat it as an error.
• Budget max_tokens from the documented context limit instead of copying a constant.
• K3.1 Pro and K3.1 Pro Highspeed are distinct model strings, so pin the ID and verify parity on your own prompts.
• Retry on rate limits with backoff, and make tool calls idempotent before they can run twice.

Source

Kimi K3.1 Pro and K3.1 Pro Highspeed announces the two variants. I added the stream-termination checks, token budgeting, and retry handling that the announcement leaves out.

Support this work

These write-ups are researched and published with no paywall, sponsor, or tracking. If one saved you an afternoon, a small tip keeps them coming.

USDT, USDC or USDD · TRC-20 (Tron)

TFTNsfyomKrnUutRjBTGVULp19ByW29KbY