Rebuilding a model's tokenizer around token cost cuts inference 77%

A team at Dharma-AI describes VocabX, a way to rebuild a small model's tokenizer around the cost of each token rather than how often it appears. Generating a token is far more expensive than reading one, because decoding is bound by memory bandwidth, and in their measurement a generated token cost 7.4 times a read token. So VocabX concentrates on cutting the tokens the model produces. It works on two levels under one cost rule: sub-word pieces for domain words, and multi-word "macro tokens" for repeated spans such as JSON envelopes. The base model, the data, and the training recipe all stay the same.

On their main production case, a Portuguese essay grader built on Qwen3-4B and run on an RTX 5090, cost fell 77 percent and the average response dropped from 350 to 101 tokens, while the benchmark score rose from 0.703 to 0.810. Across five domains and three model families, Qwen3, Llama-3.1, and Gemma-3, the method compressed more than the strongest reference approach in all 15 combinations tested. The author, Bernardo França de Aguiar, notes the results come from a paper still under review, so the numbers may change.

Why it matters

If you run a small model in production for a narrow task, this is a lever that cuts inference cost and latency without touching model quality, and here it raised accuracy as well. The gains depend on repetitive, structured output, so measure on your own traffic before expecting the full 77 percent.