Free lesson · AI Engineering

Tokenisation

Easy11 min readaillmrag

Tokenisation

The Bill That Did Not Add Up

A team at Meta running a multilingual support assistant came to my team with a puzzle. Their cost dashboards showed something that looked like a billing error: conversations in Hindi were costing roughly 3x more than conversations in English of the same length. Same feature, same model, same prompt template, same average character count per conversation. The finance-minded engineer on the team had already double-checked the provider's pricing page - one rate per million tokens, no per-language pricing anywhere. So where was the 3x coming from?

The answer was not in the pricing. It was in the counting. The provider charges per token, and a token is not a character, not a word, and not a fixed unit of anything a human would recognise. It is a chunk of text defined by the model's tokeniser - and the tokeniser, it turns out, is dramatically more efficient at chunking some languages than others. English text averaged about four characters per token. Their Hindi text was averaging closer to 1.5. Same message length in characters; nearly 3x the tokens; nearly 3x the bill. Nothing was broken. The unit of account was simply something they had never examined.

You have been living in token-world for an entire module - paying per token, budgeting context by token, truncating at max_tokens, estimating with tiktoken - while treating the token itself as an opaque unit. This topic opens that box, because the tokeniser is upstream of everything: your costs, your context limits, your latency, and a whole family of bizarre model behaviours that engineers routinely misdiagnose as model bugs.

What a Token Actually Is

Recall the one-sentence truth from the last topic: the model is a function over sequences of tokens. Before your prompt ever touches the model, a separate, completely deterministic component - the tokeniser - converts your text into a sequence of integer IDs. The model never sees text. It sees [791, 6864, 315, 9822, 374]. The response comes back as IDs too, and the tokeniser converts them back into text at the end.

So what are the chunks? The dominant approach is Byte Pair Encoding (BPE), and its logic is beautifully simple. Start with a base vocabulary of individual bytes. Scan an enormous corpus of text. Find the pair of adjacent units that occurs most frequently, and merge it into a single new vocabulary entry. Repeat - tens of thousands of times. Common sequences get merged early and become single tokens; rare sequences never merge and stay fragmented into pieces.

The result is a vocabulary of roughly 100,000 entries (GPT-4's tokeniser has ~100K; Llama 3's ~128K) where the token boundaries follow frequency in the training corpus, not linguistic logic:

" the" - one token (note the leading space; spaces attach to the following word)
" tokenization" - likely one token; it appears constantly in technical text
" Kartikeya" - probably four or five tokens; rare in the corpus, so never merged
"def " - one token; the corpus contained oceans of Python

The rule of thumb you have already been using - one token ≈ four characters ≈ three-quarters of a word - is an average over English text, and every word of that qualifier matters. It is not a property of tokens. It is a property of English's excellent representation in training corpora.

# tiktoken is OpenAI's tokeniser library - install with: pip install tiktoken
# This example shows the structure; run locally with tiktoken installed.

examples = [
    ("the", "very common word"),
    ("tokenization", "common technical term"),
    ("antidisestablishmentarianism", "rare long word"),
    ("नमस्ते", "Hindi greeting"),
    ("def fibonacci(n):", "common code pattern"),
]

# With tiktoken installed:
# import tiktoken
# enc = tiktoken.get_encoding("o200k_base")
# for text, note in examples:
#     ids = enc.encode(text)
#     pieces = [enc.decode([i]) for i in ids]
#     print(f"{text!r} -> {len(ids)} tokens: {pieces}  ({note})")

print("Run locally to see how differently these fragment.")
print("The rare word and the Hindi text will fragment into many pieces;")
print("the common word and the code pattern will be one or two tokens each.")

Run that locally - actually run it, do not just read it - and the multilingual cost mystery dissolves in front of you. The Hindi greeting fragments into several tokens where an English greeting of similar length is one or two. Scale that ratio across every message of every conversation, and there is your 3x.

The Five Practical Consequences

Tokenisation would be trivia if it stayed inside the API. It does not. It leaks into cost, capacity, latency, and capability in ways you will hit in production. Here are the five that matter, in the order they will bite you.

1. Cost Is Denominated in a Unit That Varies by Content

You know from Module 1 that you pay per token. You now know the token-per-character rate is content-dependent. The engineering consequences:

The multilingual tax is real and large. English sits around 4 characters per token on modern tokenisers. Major European languages run somewhat worse. Languages with non-Latin scripts - Hindi, Thai, Amharic, and many others - historically ran 2–4x more tokens for equivalent text, though newer, larger vocabularies (like the 200K-entry tokenisers) have narrowed this meaningfully. If you serve a global audience, your per-conversation cost varies by user language, and your cost model from Module 3 needs a language-mix assumption in it. The same applies to context: a 128K-token window holds far less Hindi than English. Your non-English users get a smaller effective context window and a bigger bill, simultaneously - a genuinely uncomfortable equity fact baked into the infrastructure.

Formats have token costs too. JSON's braces, quotes, and repeated keys; XML's verbose tags; markdown tables' pipes and dashes - every structural character is tokenised. This is the mechanism behind topic 1.3's observation that JSON output runs 20–40% heavier than prose. When you design an output schema or choose a data format for context-stuffing, you are making a tokenisation decision whether you know it or not. YAML, for what it is worth, typically tokenises lighter than JSON for the same data.

2. The Model Cannot See Characters

This is the consequence that generates the most "the model is broken" tickets, so let me be emphatic: the model has never seen a letter. By the time input reaches the model, "strawberry" is one or two integer IDs. The individual characters - s, t, r, a, w - are not in the input in any form. Asking the model how many r's are in "strawberry" is asking it to introspect the spelling of its own token IDs, which it can only do via statistical traces of spelling discussions in training data - which is why frontier models have famously, embarrassingly failed at it.

The full family of character-blind failures: counting letters, reversing strings, acrostics, rhyme precision, character-level ciphers (ROT13, base64 by hand), and "words that start with" games. None of these are reasoning failures. They are input-representation failures - the model is being asked about information that was destroyed before it arrived.

The production translation: never delegate character-level operations to generation. If your application needs string manipulation, that is Python's job, or a tool call's (Module 7). And when a stakeholder waves the strawberry example at you as proof the model "can't think," you now have the precise, boring, correct explanation.

3. Numbers Fragment Unpredictably

Tokenisers chunk digit sequences by corpus frequency like everything else: "100" might be one token, "1001" two ("100" + "1"), and "3.14159" a handful of arbitrary fragments. The model is doing arithmetic - or rather, predicting arithmetic-shaped continuations - over chunks of digits with frequency-determined boundaries. This compounds the topic 2.1 point about arithmetic unreliability: the model is bad at math partly because it cannot even see numbers in a consistent representation. Some model families now normalise digit tokenisation (single-digit tokens) precisely to reduce this chaos, but the engineering rule stands regardless: numbers that matter go to tools, not to generation.

4. Code Tokenises Differently Than You Expect

Good news first: code tokenises efficiently, because training corpora are full of it. Common keywords, idioms, and even whole patterns (for i in range() compress into few tokens. Indentation is the interesting case - runs of spaces are merged into dedicated whitespace tokens, so a deeply-nested Python file spends real token budget on its own indentation. Two practical notes: minified or unusually-formatted code tokenises worse than idiomatic code (the merges were learned from idiomatic corpora), and tabs-versus-spaces is, hilariously, a token-count question - tabs are typically one token per level while four spaces may or may not merge depending on context. Nobody should reformat a codebase over this, but when you are stuffing large amounts of code into context in the RAG modules, tokeniser-friendliness becomes a real consideration in what you send.

5. Token Boundaries Explain Prompt Edge Cases

A cluster of small mysteries dissolves once you know boundaries exist. Why does a prompt ending mid-word sometimes produce strange completions? Because you have handed the model a token fragment that rarely occurs at sequence-end in training. Why do leading spaces matter? Because " Paris" and "Paris" are different tokens with different statistics. Why did your stop sequence not fire? Because stop sequences match generated text, but generation happens in tokens, and your stop string may straddle a token boundary in a way that never gets produced as a clean unit. When prompt behaviour gets weird at the edges, look at the token boundaries before you blame the model - tiktoken shows you exactly what the model saw.

graph LR
    A["Your text:<br/>'The capital of France'"] --> B[Tokeniser<br/>deterministic, BPE merges]
    B --> C["Token IDs:<br/>[791, 6864, 315, 9822]"]
    C --> D[Model<br/>never sees characters]
    D --> E["Output IDs:<br/>[374, 12366]"]
    E --> F[Tokeniser decodes]
    F --> G["' is Paris'"]
    B -.->|"boundaries follow corpus frequency,<br/>not linguistic logic"| H[Cost, context, and<br/>capability consequences]

This diagram shows the tokenisation pipeline. Your text, such as "The capital of France", passes through the deterministic BPE tokeniser and becomes a sequence of integer token IDs. The model operates only on these IDs and never sees characters. Output token IDs are decoded back into text by the tokeniser. A dotted line notes that token boundaries follow corpus frequency rather than linguistic logic, which produces the cost, context, and capability consequences discussed in this lesson.

Working With Tokens as an Engineer

Three operational habits turn this knowledge into practice.

Count with the right tokeniser. Your topic 1.5 estimator used tiktoken with a named model - that pairing matters, because every model family has its own tokeniser. Counting GPT-4 tokens to budget a Claude context window is measuring metres with a yardstick; the counts differ, sometimes by 10–20%. For OpenAI models, tiktoken is exact. For Anthropic, use their count-tokens endpoint for precision, or accept tiktoken as a rough cross-estimate with explicit error margin. For open-source models, the tokeniser ships with the model (Hugging Face's AutoTokenizer), and using the right one is a one-line change that removes all guesswork.

Budget in tokens, not characters, from the start. Every threshold in your application - context compaction triggers, document chunk sizes (a decision with enormous consequences waiting in Module 5), input validation limits - should be denominated in tokens of the target model. Character-based budgets are wrong by a content-dependent factor, and "content-dependent factor" is another way of saying "works in English testing, breaks for your international users."

Make token counts visible in development. The single best habit from this topic: during development, log not just usage totals but occasionally look at the actual tokenisation of your prompts. Ten minutes with a tokeniser visualiser on your system prompt will routinely find 15–20% waste - verbose JSON where compact prose would do, decorative headers, redundant instructions - and topic 16.1 will industrialise that instinct into a discipline.

Interview: A production-debugging interview favourite: "Your per-request costs for a global chat product vary 3x by region with identical usage patterns. Diagnose." The expected path runs straight through this lesson: costs are per-token, tokens-per-character varies by language, non-Latin scripts fragment heavily, so regional language mix drives regional cost - and the strong candidate adds the equity kicker (smaller effective context for the same users) and a mitigation or two (tighter prompts, model/tokeniser choice, summarisation cadence tuned per language). Candidates who have never looked inside a tokeniser cannot find this; it looks exactly like a billing bug.

Exercise: Meet Your Tokeniser

Exercise: Install tiktoken and spend thirty minutes looking at what the model actually sees:

Tokenise your Module 1 system prompts. Compute characters-per-token for each. Find and eliminate your three most token-expensive phrasings - you have just done your first prompt-cost optimisation with evidence.

Tokenise the same paragraph in English and one other language you know. Compute the ratio. If you serve multilingual users, put that ratio into your Module 3 cost model - for real.

Tokenise "strawberry", then ask your model to count its r's at temperature 0, five times. Connect what you see in the token IDs to what you see in the answers.

Tokenise the same small JSON object as JSON, as YAML, and as a compact prose sentence. Rank by token count. Now re-read your topic 1.3 schema with new eyes.

Challenge: take a stop sequence you used in Module 1 and check whether it tokenises as a clean unit or straddles boundaries in realistic contexts. Explain to yourself why one is more reliable than the other.

The Tokens You Never Typed: Special Tokens and Chat Templates

One last layer of the box, because it quietly reframes something you have used since topic 1.1. The vocabulary does not only contain text chunks - it contains special tokens: reserved entries that never appear in normal text, used as structural markers. The EOS (end-of-sequence) token you will meet properly in topic 2.4 is one. The others that matter to you are the chat markers.

When you send a messages array - system, user, assistant - the provider does not hand the model your JSON. A chat template flattens the conversation into one long token sequence, with special tokens delimiting the roles: something like <|im_start|>system ... <|im_end|> <|im_start|>user ... <|im_end|> (the exact markers vary by model family). The model learned, during training, that instructions between system markers govern behaviour and that generation should follow an assistant marker. The entire roles abstraction of Module 1 is, underneath, a tokenisation convention.

Three practical consequences. First, the template costs tokens - a handful per message, which is the per-message overhead your topic 1.5 estimator approximated with + 4. Second, role separation is learned, not enforced: nothing architecturally prevents text in a user message from resembling instruction patterns, which is the mechanical seed of prompt injection - Module 4 builds the defences, but the vulnerability is visible from here. Third, when you self-host open models (Module 16), applying the correct chat template is your job, and using the wrong one is a classic silent-degradation bug: the model still responds, just noticeably worse, because the structural markers it trained on are missing or malformed.

What You Should Take From This Lesson

A token is a frequency-defined chunk of bytes, built by BPE merges over a training corpus, and it is the true unit of everything you pay for and everything the model perceives. The four-characters rule is an English average, not a law - languages, formats, numbers, and code all tokenise at different rates, which makes cost and context capacity content-dependent in ways your budgets must respect. The model is character-blind by construction: letter-counting failures are input destruction, not reasoning failure, and character-level work belongs in code, not generation. Count with the model's own tokeniser, budget in tokens from day one, and look at real tokenisations of your prompts - the waste you find will pay for the time.

You now know what goes into the function: sequences of frequency-chunked integer IDs. The obvious next question is what happens inside - how does a pile of frozen numbers turn those IDs into a distribution that seems to understand context, reference, and relevance? The answer is one mechanism, repeated in stacked layers, and it is called attention. That is the next topic - and I promise you the payoff is practical: attention is why input tokens are cheaper than output tokens, why prompt caching works, and why your long conversations degraded in exactly the way topic 1.5 described.

Back to AI Engineering