Switching on constrained decoding took vLLM tool calls to 100%

Ubicloud's engineers hit a common failure while serving Kimi K2.6 on vLLM: the model produced a valid tool-call schema only 92% of the time, so about one call in twelve was malformed. Those bad calls are not harmless. They pollute the context window, drag down later output, and cost money through retries. The root cause is simple. A language model samples tokens by probability and never checks its output against a schema, and training data from other tool-calling setups can push it toward names that don't exist in your environment, such as search_web when the tool is actually called web_search.

There was a second, sneakier cause. When Ubicloud moved to upstream vLLM's constrained decoding in v0.25, accuracy slipped to 91%, because vLLM had quietly changed the default so constrained decoding was off unless a request asked for it. The team explains that vLLM serves many models and constrained decoding has per-model quirks, so turning it off by default was a reasonable call for the project even though it surprised users.

The fix is constrained decoding itself. At each step vLLM uses XGrammar structural tags to mask out any token that would break the call's structure, so a valid schema becomes a runtime guarantee rather than a hope, and the authors say the overhead is negligible because the masks are precomputed. Turned back on and tested on the K2VV ToolCall benchmark, schema accuracy reached 100% across every model version Ubicloud serves, which it has now deployed for its Kimi and DeepSeek endpoints.

Why it matters

If you run open models for agent workloads, do not assume valid tool calls come for free: check whether your vLLM build has constrained decoding enabled, because the default may be off and a 92% success rate quietly breaks one agent step in twelve.