Tokens and Tokenizers in LLMs: Why Does Turkish Take More Tokens?

Version note: The examples in this post were tested with Ollama 0.34 on local models (gemma3:4b, qwen3:4b), using .NET 10 and Node 22.

Guess First

Bookstore sends its customers a notification: "Your order has been shipped and will arrive tomorrow." The Turkish version of the same sentence is "Siparişiniz kargoya verildi ve yarın elinizde olacak." The Turkish sentence has 7 words, the English one 9. Which one is cheaper for the model to read? And how many tokens is a single word like "okuyamadıklarımızdan" ("from the ones we could not read"): one, five, ten?

Write your guess down. In this post we will actually count tokens: first Turkish versus English, then code, numbers and IDs. Most of the answer depends on the tokenizer of the model you use, and we will see that with two different models.

What Is a Token?

A model reads text neither letter by letter nor word by word; it reads it token by token. A token is a piece from the model's vocabulary. Common words are usually a single token, while rare words are split into several pieces. The component that builds this vocabulary and splits text into pieces is called the tokenizer.

Most of today's tokenizers use a subword approach. The best known is Byte-Pair Encoding (BPE): it starts from characters, merges the pair of pieces that appear next to each other most often and repeats this until the vocabulary reaches the desired size (Sennrich et al., 2015). SentencePiece is a language-independent tokenizer trained directly on raw text that treats whitespace as an ordinary symbol (Kudo and Richardson, 2018). Gemma 3 uses SentencePiece; according to its technical report, its vocabulary has 262 thousand tokens, it splits digits one by one and it falls back to bytes for characters that are not in the vocabulary.

An analogy: a tokenizer works like a typesetter splitting text into syllables. But it picks the syllables not by grammar but by how often it saw them in the training data. If the training data contains a lot of English, English words split into a few large pieces, and less common languages into many small ones.

Counting Tokens: Ollama's Counters

Ollama 0.34 has no separate tokenize endpoint; a request to /api/tokenize returns 404. But every response carries prompt_eval_count. If we send the prompt with raw: true, so it does not go through the chat template, and ask for a single token, that number is almost exactly the token count of the text:

import ollama from "ollama";

// raw: true skips the chat template; only the tokens the model adds at the start get mixed in
export async function countTokens(model: string, text: string): Promise<number> {
  const res = await ollama.generate({
    model,
    prompt: text,
    raw: true,
    options: { temperature: 0, num_predict: 1 },
  });
  return res.prompt_eval_count;
}

Why "almost"? Gemma 3 adds a beginning-of-sequence (BOS) token at the start of every text. When I counted the word "Hello" with this function, I got 2 for Gemma 3 and 1 for Qwen 3. So each Gemma number below includes that one-token share. What matters when comparing is counting with the same method on the same model.

Turkish and English: Same Sentence, Different Bill

The code in this section contains Turkish text because its topic is Turkish itself. We count four translation pairs with the tokenizers of two different models:

import { countTokens } from "./count-tokens.js";

const pairs: [string, string][] = [
  ["Your order has been shipped and will arrive tomorrow.",
   "Siparişiniz kargoya verildi ve yarın elinizde olacak."],
  ["Can you recommend a short science fiction novel for a long train journey?",
   "Uzun bir tren yolculuğu için kısa bir bilim kurgu romanı önerebilir misiniz?"],
  ["We could not find any books matching your search.",
   "Aramanızla eşleşen hiçbir kitap bulamadık."],
  ["from the ones we could not read",
   "okuyamadıklarımızdan"],
];

for (const model of ["gemma3:4b", "qwen3:4b"]) {
  console.log(model);
  for (const [en, tr] of pairs) {
    const enTokens = await countTokens(model, en);
    const trTokens = await countTokens(model, tr);
    const enWords = en.split(" ").length;
    const trWords = tr.split(" ").length;
    console.log(`  EN tokens=${enTokens} words=${enWords} | TR tokens=${trTokens} words=${trWords}`);
  }
}

gemma3:4b
  EN tokens=11 words=9 | TR tokens=16 words=7
  EN tokens=15 words=13 | TR tokens=23 words=12
  EN tokens=11 words=9 | TR tokens=13 words=5
  EN tokens=8 words=7 | TR tokens=7 words=1
qwen3:4b
  EN tokens=10 words=9 | TR tokens=21 words=7
  EN tokens=14 words=13 | TR tokens=25 words=12
  EN tokens=10 words=9 | TR tokens=16 words=5
  EN tokens=7 words=7 | TR tokens=8 words=1

The answer to the opening question: on Gemma 3 the shipping sentence is 11 tokens in English and 16 in Turkish. The Turkish sentence is two words shorter and still about 45 percent more expensive. In the second pair the gap is 53 percent, in the third 18 percent. "okuyamadıklarımızdan" alone takes 7 tokens, almost the same as its seven-word English equivalent.

Reading this table per word would be misleading. Turkish spends 2–3 tokens per word, English around 1.2. But Turkish packs more meaning into each word with suffixes, so the gap per sentence is not as large as the gap per word. The real surprise is the second model: with the Qwen 3 tokenizer the same Turkish sentences take between 1.6 and 2.1 times as many tokens as the English ones (21 vs 10, 25 vs 14, 16 vs 10). The Gemma 3 technical report says its new tokenizer is more balanced for non-English languages, and the measurement supports that.

The literature points the same way. The NeurIPS 2023 study by Petrov et al. shows that token length can differ by up to 15 times between translations of the same text. A 2025 study on Turkish (Bayram et al.) reports that how well a tokenizer captures valid Turkish pieces correlates with the model's Turkish MMLU results. The practical takeaway for you: do not estimate the token budget of a Turkish application from English examples; measure it on the model you will use.

Code, Numbers and IDs

What you send an LLM is not always a well-formed sentence. The Bookstore assistant will also see order numbers, JSON, code snippets and IDs. Let's count those too:

import { countTokens } from "./count-tokens.js";

const samples = [
  "1234567890",
  "1,234,567.89",
  "2026-11-06",
  "3f2504e0-4f89-11d3-9a0c-0305e82c3301",
  '{"orderId": 1042, "status": "shipped"}',
  "var total = order.Items.Sum(i => i.Price * i.Quantity);",
  "return total;",
  "        return total;",
];

for (const text of samples) {
  const tokens = await countTokens("gemma3:4b", text);
  console.log(`${String(tokens).padStart(3)} tokens  ${String(text.length).padStart(3)} chars  ${text}`);
}

 12 tokens   10 chars  1234567890
 14 tokens   12 chars  1,234,567.89
 12 tokens   10 chars  2026-11-06
 38 tokens   36 chars  3f2504e0-4f89-11d3-9a0c-0305e82c3301
 18 tokens   38 chars  {"orderId": 1042, "status": "shipped"}
 20 tokens   55 chars  var total = order.Items.Sum(i => i.Price * i.Quantity);
  4 tokens   13 chars  return total;
  5 tokens   21 chars          return total;

Numbers cost more than one token per character: because Gemma 3 splits digits one by one, a ten-digit number takes 12 tokens. That is also why the year 1949 arrived as four separate tokens in the loop from the first post. The most expensive one is the GUID: 38 tokens for 36 characters. Anything that looks random (GUIDs, hashes, tokens, base64) finds no match in the tokenizer's vocabulary and gets split almost character by character. Remember that before you put hundreds of order IDs into a prompt.

Code, on the other hand, is not as expensive as people think. The C# line takes 20 tokens for 55 characters, because pieces like return, total and = show up very often in training data. Indentation is cheap too: eight leading spaces added just one token. JSON looks similar; key names and quotes go as short pieces.

The Hidden Tokens of the Chat Template

So far we counted with raw: true. In a real chat request, Ollama places your message into the chat template, and that template has tokens too. Let's look through Usage on the C# side:

using Microsoft.Extensions.AI;
using OllamaSharp;

IChatClient client = new OllamaApiClient(new Uri("http://localhost:11434"), "gemma3:4b");
ChatOptions options = new() { Temperature = 0, Seed = 42 };

string[] prompts =
[
    "Hello",
    "Your order has been shipped and will arrive tomorrow.",
    "Summarize the plot of Dune in exactly three sentences.",
];

foreach (string prompt in prompts)
{
    ChatResponse response = await client.GetResponseAsync(prompt, options);
    // Generation time is Ollama-specific and arrives in AdditionalProperties as a TimeSpan
    TimeSpan evalDuration = (TimeSpan)response.AdditionalProperties!["eval_duration"]!;
    long output = response.Usage!.OutputTokenCount!.Value;

    Console.WriteLine($"input={response.Usage.InputTokenCount} output={output} " +
        $"generation={evalDuration.TotalMilliseconds:F0} ms  \"{prompt}\"");
}

input=11 output=48 generation=2117 ms  "Hello"
input=20 output=37 generation=1658 ms  "Your order has been shipped and will arrive tomorrow."
input=21 output=145 generation=3350 ms  "Summarize the plot of Dune in exactly three sentences."

"Hello" was 2 tokens raw and became 11 in a chat request. The shipping sentence went from 11 to 20. In both cases the difference is 9 tokens: the template pieces that mark who is speaking and that it is the model's turn. Nine tokens look negligible in a single message, but in a ten-turn conversation every message pays that share again, and the system prompt is sent from scratch with every request.

The output side is even more instructive. Saying "Hello" made the model write a 48-token greeting. The three-sentence Dune summary took 145 tokens and was the longest request. What really drives latency is how much the model writes. Timings jump around on this machine because of other jobs running at the same time: the first two requests were generated at 22–23 tokens per second, the Dune request at 43.

Why Tokens Matter

There are three reasons: cost, limits and speed. Cloud providers price input and output tokens separately, usually per million tokens. Output tokens are mostly more expensive than input, and some providers discount repeated prompt parts through caching. Prices change often, so I am not quoting numbers here; we will do the cost math in the sixteenth post. But the formula is always the same: if the same content takes 45 percent more tokens in Turkish, the bill grows by the same amount.

On the limit side, the number of tokens a model can see in one request is limited; this is called the context window, and it is the topic of the next post. The limit is measured in tokens, not characters. So a Turkish document hits the limit earlier than an English document of the same length.

On the speed side there are two separate jobs. Processing the prompt is a fast job done in bulk: in the long-prompt experiment of the next post, according to Ollama's log, 2051 tokens were processed in 2.3 seconds, about 880 tokens per second. Generating the answer moves token by token: the 18-token answer of that same request was generated at about 21 tokens per second. A long answer keeps you waiting much longer than a long prompt.

Common Pitfalls

The first pitfall is trusting the "4 characters are about 1 token" rule. It is a rough estimate for plain English text. It does not hold for Turkish text, numbers or IDs; the GUID above took more than one token per character.

The second is using one model's token count for another model. Tokenizers are model-specific. The same Turkish sentence took 16 tokens on Gemma 3 and 21 on Qwen 3. When you switch models, redo the budget math.

The third is counting only the user's message. The real input also includes chat template tokens, the system prompt, the chat history and, as we will see later, tool definitions. The bill is set by prompt_eval_count or Usage.InputTokenCount, not by the character count in your text box.

The fourth is believing the cache lowers the token count. In the second post, even when 15 of 20 tokens came from the cache, prompt_eval_count stayed at 20. The cache shortens the time; Ollama still reports the whole prompt. Cloud providers may report cached tokens separately and at a discount; check the provider's documentation.

A Small Experiment

★☆☆ Add your own Bookstore sentences to tr-vs-en.ts: a return policy, a book blurb. See how much the gap between Turkish and English changes from sentence to sentence.

★★☆ With code-tokens.ts, count the same JSON once indented and once on a single line. Then shorten the key names (id instead of orderId) and measure the gain. How much would that difference be worth in an application that sends large JSON to an LLM?

★★★ Add a prompt to the C# example that asks the same question in Turkish and request a Turkish answer. Compare the output tokens and generation time, along with the input, with the English answer.

In the next post we will look at the space these tokens have to fit into: the context window. We will see with a real experiment that the model actually remembers nothing, that you send the chat history yourself every time, and how Ollama's default 4096-token window silently cuts a long prompt without any error.