Kimi K2 Thinking 在 15 個服務商的工具調用 schema_accuracy 介乎 84.63% 至 100%

Two rows out of a table Moonshot publishes in their K2 Vendor Verifier repo:

Same weights. Same 4,000-request test set. Same reference to compare against (Moonshot's own endpoint decides what a correct response looks like). The only variable is who served the model.

Moonshot built that harness after a run of "K2 is bad at tool calling" reports that turned out to be deployment problems. Their write-up is direct about why: the more open the weights, the more channels they get deployed through, and the less control anyone has over the result. From the outside you can't tell a model defect from an engineering deviation, and if users can't tell, the model gets blamed for both.

The two metrics measure different failures

tool_call_f1 scores the decision to call a tool at all. It's computed off finish_reason: true positive when the deployment and the reference both return tool_calls, false positive when only the deployment does, false negative when only the reference does. It answers one question. Does this endpoint decide to call tools at the same moments the official API does.

schema_accuracy is count_successful_tool_call / count_finish_reason_tool_calls. It only counts responses that already committed to a tool call, then asks how many produced arguments that validate against the JSON schema.

The second metric is the one that reaches your error handler. finish_reason says tool_calls, your dispatcher pulls the arguments out, the payload doesn't match the schema, and you get something like this in the middle of a 40-step run:

jsonschema.exceptions.ValidationError: 'path' is a required property
  at tools.read_file(arguments={'file': 'src/api/handlers.ts'})

A retry sometimes papers over it. It costs you a round trip and a chunk of context either way, and in a loop that is already deep into a plan, the retry happens without the model knowing why.

The spread, on identical weights

Kimi K2 Thinking, 15 deployments, temperature 1.0:

Sixteen points between the top and the bottom of the same model. On the K2 0905 table the floor is lower: five deployments at 100%, then vLLM at 76.00%, SGLang at 73.13%, Volc at 72.86%, Baseten at 72.49%, Together at 71.96%. Roughly 28 points of spread on identical weights.

A perfect schema score can still be a broken deployment

Look at Groq on the 0905 table: 100.00% schema accuracy, and a trigger F1 of 69.52%. The schema metric only looks at calls the model already decided to make, so a deployment that fires tool calls at the wrong moments keeps a clean schema score. Nebius shows the same shape, 84.47% schema with 50.60% F1. Read the pair together or the number lies to you. High schema plus low F1 means the endpoint is confident about when to call tools and wrong about it.

Per-call accuracy is the wrong unit

Schema accuracy is a per-call number. Your agent doesn't make one call, it makes a session's worth, and every call has to validate. If the per-call probability is p, the chance that all N calls in a session validate is p^N.

A 1.04 point gap between SiliconFlow and Fireworks looks like noise per call. Over a 25-call session it's a 23 point gap. This is the same reason a cost-per-PR number is a distribution rather than a price: the interesting behavior lives in the tail of the repeated event, not in the average of one.

Two honest caveats on the table above. The denominators differ per provider (Moonshot's official run triggered 1958 tool calls, Together's 1893, Chutes' 3657), so these are not perfectly paired samples. And a schema failure in production is often a retry rather than a dead run, so p^N is a lower bound on session success, not a prediction of crash rate. The shape holds either way: small per-call differences get amplified by repetition, and the amplification is where agents live.

The same provider shows up at both ends

DeepInfra is 100.00% on k2-0905-preview at temperature 0.6, and 86.91% on k2-thinking at temperature 1.0. Together is 71.96% on one table and 84.63% on the other. So a provider's score is a property of a specific model version, decoding parameters, and serving stack, not a brand fact you can carry forward. KVV's first fix after the initial reports was exactly this class of thing: enforcing Temperature=1.0 and TopP=0.95 in thinking mode, and requiring that thinking content be passed back to the API.

The repo also names two failure modes that are worth reading before you file a bug against a model. Tool call IDs have to follow the format functions.func_name:idx, and Moonshot's own API rewrites incoming IDs into that shape automatically. A provider that doesn't rewrite them feeds the model malformed IDs from history and gets parsing failures back. And there's no built-in schema enforcement: the model generates tokens by probability, so the README tells vendors to add guided encoding rather than trust prompting. Their recommended versions are vLLM v0.11.0 or newer and SGLang v0.5.3rc0 or newer for 0905, with v0.11.1rc6 and v0.5.5.post2 for thinking.

If you route through an aggregator, you're already measuring a moving target

Most teams don't pin a provider. They point at a router and take whatever answers. OpenRouter's default is price-based load balancing across providers with allow_fallbacks set to true, so the endpoint that serves your request can change between runs without any config change on your side. Their docs are explicit that a sort or order preference disables load balancing.

That means your eval score, your p95 latency, and your schema accuracy are all attached to a provider you didn't choose. For an eval run, pin it:

extra_body={"provider": {"only": ["fireworks"], "allow_fallbacks": False}}

If you run a reviewer or an agent on your own keys, the provider is a config field you own rather than a default you inherit. Kodus's BYOK setup has that shape, keys in and a model choice per task, and it's the right shape for this problem: the provider is a variable you set per job and can hold constant while you measure something else.

A protocol you can run in an afternoon

1. Pick one model version and one decoding parameter set. Write both down, because the next table you compare against won't have them.
2. Run your own tool-call test set against at least two providers, and log finish_reason plus a schema validation result per response. Do not log only pass or fail on the session, or you'll lose the per-call number.
3. Report schema_accuracy next to trigger F1. A clean schema score on its own has already been shown to hide a deployment that calls tools at the wrong times.
4. Re-run after every version bump on either side, and treat a provider change as a model change for eval purposes.

The KVV table is a deployment check, not a model review. The useful takeaway isn't which vendor is on top this month, it's that identical weights produce a 16 to 28 point spread in the number your agent depends on, and that spread is something you can reproduce on your own endpoint in an afternoon. If you can't reproduce it, you don't have a number. You have a vendor's number.