reasoning_effort 為何在不同服務供應商間表現不同

📝 Originally published (in Japanese) at forge.workstyle.tech.

TL;DR The app uses OpenAI-compatible /v1/chat/completions endpoints across cloud, self-hosted vLLM, and a relay. Browser actions never ran because an error was classified as lack of tool support. The cause was provider-specific handling of reasoning_effort and a classifier that ignored the error details.

① The Cloud Rejected the Combination of Tools and Configuration

The first rejection occurred when we sent the tools and reasoning_effort simultaneously. The target endpoint was gpt-5.6-luna. We received a 400 status code with the following message:

Function tools with reasoning_effort are not supported for gpt-5.6-luna in /v1/chat/completions. To use function tools, use /v1/responses or set reasoning_effort to 'none'.

Upon seeing tool and not supported in the error body, the application concluded that "this model does not support tools." From then on, it excluded tools from requests entirely.

This was the erroneous misjudgment introduced at the beginning. Even when asked to perform browser operations, no tools were provided, so not a single operation was executed. Although it might appear that the model was avoiding calling tools, there was simply no opportunity to call them in the first place.

The error message explicitly provided a solution. To use tools, one should either use /v1/responses or set reasoning_effort to 'none'. The application, however, skipped reading that part and immediately made the decision to discard the tools.

The fix was to read the resolution before treating the rejection as a permanent tool incompatibility. We reissued the request with reasoning_effort: 'none' while keeping the tools attached. Essentially, for this model, using tools meant stopping the reasoning process.

Speed-wise, this change had its appeal. Setting reasoning_effort: 'none' reduced the latency from input to the start of the output from 4–5 seconds to just 2 seconds.

② Our in-house vLLM flat-out rejected the value

When I applied the same configuration to the tool-calling path, our own vLLM instance returned a 400 error.

Input should be 'low', 'medium' or 'high'

The error type was literal_error. It wasn't that the field was missing; rather, the endpoint simply did not accept the value none.

If you only test with plain conversation, you might stop right here. Since conversations speed up, it looks like things are working. It was only after testing the tool-calling path that we realized the endpoint would handle values differently.

For this endpoint, we detect the value rejection and stop sending reasoning_effort altogether. To disable thinking, we instead send chat_template_kwargs: { enable_thinking: false }.

Here are the measured results for comparison.

Method of disabling thinking	Time taken
Do nothing	87 seconds
`/no_think`	154 seconds
`enable_thinking=false`	65 seconds
`reasoning_effort none`	2 seconds

These figures were measured using a model on-device. The differences between methods were significant, meaning that just because configuration names are similar does not imply the behavior will be aligned.

On top of that, the output fields for thinking don't match up either. The name used varies among three possibilities: reasoning, reasoning_content, and thinking. The writing side was aware of all three, but the reading side was only looking for reasoning.

vLLM writes to reasoning_content. The moment we switched to our own vLLM, the classifier flagged "empty response" five times in a row. The model was returning the thinking, but because the reader wasn't looking at the correct field, we couldn't even count the amount of thinking generated.

③ The relay server returned empty only when streaming

The third point concerns the handling of sequential responses.

There were some endpoints that ignored the stream: true flag and returned standard JSON. The app discards any line that does not start with data:. As a result, zero fragments were captured. Since response.ok was true, no exception was thrown, and nothing appeared on the screen.

I fixed this by checking the content-type to determine whether the response is a streaming response or standard JSON.

But that wasn't the end of it. One relay server endpoint would return a 200 OK in 94ms only when streaming was enabled, but the body was empty. It would reject non-streaming calls with a 400 error. In empty responses, there are no clues like an error message body.

For this endpoint, I decided to treat an empty response as an exception. I configured it to retry once after disabling the "thinking" specification, and to stop sending requests after that. When I retried without the specification, the correct JSON was returned in 327ms.

A table of measured behavior per endpoint

Endpoint behavior	Data sent
Standard (OpenAI, etc.)	`reasoning_effort: 'none'`
`none` not in the value list (vLLM, rejects with 400)	`chat_template_kwargs: { enable_thinking: false }`
Returns empty when streaming (relay server)	Send nothing

All of these were decided based on actual measurements. I don't ask the users. The app learns based on the response from the endpoint.

Self-Inflicted Regressions

After implementing a fix to account for differences between endpoints, I introduced a couple of new issues myself.

Regression 1: Overly Broad Handling of 400 Error Bodies

The logic for detecting rejected values was also checking for the presence of the word invalid. If a 400 error body contained invalid_request_error, all errors were treated the same.

As a result, the system incorrectly learned that gpt-5.6-luna was an endpoint "missing the reasoning_effort field". It stopped at the none designation required in ①, causing the entire browser operation to fail.

The root cause was that I had created custom error messages for testing. Since these differed slightly from actual rejections, the discrepancy in detection went unnoticed. I adjusted the tests to use recorded error bodies directly.

Regression 2: Applying Local Settings to Cloud Endpoints

In voice conversations, I had enabled a setting to automatically truncate thoughts. This setting was inadvertently applied to the official cloud endpoints as well, causing none to be sent to OpenAI.

Every response generation request timed out after 60 seconds, resulting in a 500 error. This occurred consistently across four attempts, despite OpenAI's status being "normal".

The code originally noted that "cloud providers use the same keys for different purposes". I failed to consider endpoint-specific meanings and applied the setting globally. I corrected this by limiting the setting to local providers only.

Additionally, there was a flaw in the retry mechanism for rejected fields. Each utterance allowed only one retry. If both temperature and frequency_penalty were rejected, the first utterance would fail silently.

I updated the logic to remove and retry rejected fields upon each rejection, up to 8 attempts. If a response with at least one character is received before a failure, no retry is attempted.

Key takeaways

Read the full 400 response body before classifying the error.
Match reasoning_effort handling to each endpoint's observed behavior.
For vLLM, use chat_template_kwargs: { enable_thinking: false } when none is rejected.
If a relay returns an empty streaming response, retry without the reasoning setting and stop sending it to that endpoint.
Use observed endpoint behavior in tests and provider-specific configuration.