Polski · interactive report / TRAVNY

AI Token Worldcup: 27 languages, six tokenizers

In this sample, Chinese uses 89 GPT-5 tokens; English uses 95. Polish / English token counts: GPT-5 (o200k_base): 123 / 95; Claude Opus 5.5: 199 / 128; DeepSeek V4.1: 141 / 95.

Results

Input text tokens, sorted by GPT-5 count
LanguageGPT-5 (o200k_base)Claude Opus 5.5DeepSeek V4.1cl100k_baser50k_baseo200k_harmony
Chinese891178612121389
English9512895958395
Dutch9918712412713399
Norwegian104153128129131104
Swedish104155126125133104
Spanish106161117117121106
French107166124124128107
German108204126131153108
Turkish108197152144163108
Vietnamese108223161183295108
Portuguese109187124122140109
Indonesian110212114132163110
Arabic111212126205297111
Hebrew112207132279321112
Russian114154125167349114
Latin118189128129128118
Italian120181125136148120
Finnish121214142159156121
Polish123199141152190123
Ukrainian130170157201370130
Hindi133275214384550133
Korean134207146196379134
Slovak135182149160190135
Czech136160140166200136
Japanese137156129187250137
Hungarian141176143153165141
Greek149260186323397149

Method

Each language conveys the same neutral facts about balls, arithmetic and equal division. Samples are natural formulations, not word-for-word translations. GPT-5 uses o200k_base via gpt-tokenizer; DeepSeek uses its official pinned V4.1 tokenizer. Claude Opus 5.5 counts come from Anthropic count_tokens with eight framing tokens subtracted. The three additional GPT encodings are cl100k_base, r50k_base and o200k_harmony.

Limitations

One short sample per language, not a representative corpus. Counts depend on wording, script and tokenizer. They are not a general language ranking, a measure of model quality, or a prediction of total API cost.

Reproduce and cite

Download the exact corpus and token counts (JSON) · Quarto source, recount scripts and raw measurement files · Interactive report and charts.