The sticker price of an LLM hides what its tokenizer charges you

Comparing model prices by the advertised dollars per million tokens misses a hidden variable: the tokenizer. The same text becomes a different number of tokens depending on the model, so two models with the same sticker price can cost very different amounts to run. In one test, a single TypeScript file came to 681 tokens on GPT-5.x but 1,178 on Claude's new tokenizer, 73% more for identical code.

The gap is worse for code than for prose. Against OpenAI's o200k baseline, Claude's newer tokenizer produced about 1.40x the tokens on English, 1.58x on Rust, and 1.73x on TypeScript. There is also a trap inside a single vendor. Claude Opus 4.6 and 4.8 carry the same $5 and $25 per million pricing, yet 4.8's new tokenizer emits roughly 32% more tokens for the same content, so the real cost of Opus 4.8 works out closer to $7.50 and $37.50 per million once you account for the inflation. The author measured this with official token-counting endpoints across 16 content types, then confirmed the figures against real billed API calls that cost about $0.08.

Why it matters

If you budget or compare LLM APIs, especially for coding agents that push large volumes of code through the context, the list price can understate your real bill by a third or more. Measure token counts on your own content with each vendor's tokenizer before you choose, because the cheaper sticker is not always the cheaper model.