AlgoInsight

Q&A

What is tokenization in NLP?

By AlgoInsight Team2026-06-18

Overview

A neural network processes numbers, not text. Tokenization converts text to a sequence of token IDs that are looked up in an embedding table. The design of the tokenizer directly impacts model capabilities - vocabulary size determines the embedding table's size; token boundaries determine what "one step" of the model processes.

graph LR
    A["Raw text: 'unhappiness'"] --> B["Tokenizer"]
    B --> C["Tokens: ['un', 'happ', 'iness']"]
    C --> D["IDs: [1234, 567, 890]"]
    D --> E["Embedding lookup → vectors"]
    E --> F["LLM processes token sequence"]
    style B fill:#1e1b4b,color:#fff

Tokenization Strategies

Character-level

Each character is a token. Very long sequences; simple vocabulary; models struggle to learn word-level meaning across long spans.

Word-level

Each word is a token. Manageable sequences but huge vocabulary (millions of words); can't handle new words or morphological variants ("running" and "runner" are unrelated).

Subword tokenization (standard for LLMs)

Decompose words into sub-units - common substrings get their own token; rare words are split into pieces. Balances vocabulary size (~32k–128k tokens) and sequence length. Three main algorithms:

BPE (Byte-Pair Encoding) - GPT-2, GPT-3/4, LLaMA: Start with characters; repeatedly merge the most frequent adjacent pair into a new token. The vocabulary is the set of all merged tokens.

WordPiece - BERT: Similar to BPE but merges pairs that maximize likelihood of the training data.

SentencePiece - LLaMA, T5, multilingual models: Language-agnostic; treats the raw text stream (bytes) directly, works well across scripts.

Tokenization Examples

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("gpt2")

tokenizer.encode("Hello, world!")
# [15496, 11, 995, 0]  - 4 tokens

tokenizer.encode("unhappiness")
# [403, 34191, 1108]   - 3 tokens (un / happi / ness, approximately)

tokenizer.encode("ChatGPT")
# [38, 265, 38, 11571]  - 4 tokens (subword split)

Why Tokenization Explains LLM Quirks

LLM failures that seem mysterious often reduce to tokenization:

Counting letters

Q: How many 'r's in "strawberry"?

The LLM may fail because "strawberry" can be a single token or a few tokens - the model never "sees" individual letters unless they're individual tokens.

Spelling/reversals

Letter-level tasks are hard because the model operates at the token level, not character level.

Arithmetic struggles

"100" might be one token, "99" might be two tokens ("9" + "9") - the model doesn't reason about digit positions directly.

Non-English performance gaps

English subwords appear in training data far more than other languages → better compression for English (fewer tokens per word) → more model capacity dedicated to English patterns.

Context Window and Tokens

LLMs' context window limits are measured in tokens, not words or characters. Rule of thumb: ~1 token ≈ 0.75 words ≈ 4 characters (for English). A 128k-token context window ≈ ~96,000 English words ≈ ~200 pages.

Tiktoken (OpenAI's tokenizer library)

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")   # GPT-4 tokenizer
tokens = enc.encode("Hello, world!")
print(len(tokens))    # 4
print(tokens)         # [9906, 11, 1917, 0]

Why This Matters in Interviews

Tokenization is foundational but often overlooked. Interviewers test: the three tokenization approaches (character / word / subword) and why subword wins, BPE/WordPiece/SentencePiece, that "context window" is measured in tokens (not words), and that tokenization explains concrete LLM limitations (counting, spelling, multilingual performance). Showing you understand that the model never directly sees characters unless they're individual tokens grounds the interview in practical depth.

Common Pitfalls

References

More answers