CLEAR's tokenizer study shows why equal token prices can produce unequal language costs

Research resurfacing in October coverage measures translated content across roughly 200 languages, with the worst max-to-min gap reaching seventeenfold.

A September 29 study by the Coalition for Language Equity in AI Research, highlighted again in October news coverage, examines a cost difference hidden by uniform per-token pricing. Equivalent content in different languages can require very different numbers of tokens from the same model.

CLEAR tokenized professionally translated FLORES-200 sentences across roughly 200 languages. Its maximum-to-minimum token-count ratio ranges from 2.7 for the most equitable tokenizer studied to 17 in the worst case. These are comparisons between the most and least efficiently tokenized languages within each evaluated system, not a claim that every non-English request costs seventeen times more.

A concrete example illustrates the mechanism. Using the study's GPT-5 tokenizer example, equivalent sentences required 13 tokens in English, 26 in Malayalam and 78 in Amharic. Fragmenting less familiar scripts into smaller units can make a short statement consume substantially more of a model's context.

The study also shows why one ranking cannot describe every aspect of language support. A model may serve most languages relatively efficiently while performing poorly on a small number. CLEAR therefore reports both worst-case ratios and distributional measures, alongside the proportion of languages close to a model's cheapest language.

For multilingual products, the implication reaches beyond the invoice. A conversation history can fill the available context faster in one language, while generating the response requires more steps. Comparing equivalent tasks across supported languages makes those differences visible. Tokenizer efficiency belongs alongside answer quality when assessing whether a service offers a comparable experience to its users.

Sources

- [The Tokenizer Tax](https://clear-languages.org/blog/the-tokenizer-tax) — Coalition for Language Equity in AI Research