Tokenisation: How Text Becomes Numbers

§ 01 · HookWhy Tokenisation Matters

Open a Python terminal and run this:

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
enc.encode("ChatGPT is great!")
# → [16047, 38, 2898, 374, 2294, 0]

Six numbers. That's what a GPT model using the cl100k_base encoding actually receives when you type "ChatGPT is great!". It doesn't see text or letters, just a sequence of integers. And notice that there are six of them for three words and an exclamation mark. Decode them one at a time and you get 'Chat', 'G', 'PT', ' is', ' great', '!'. The name "ChatGPT" takes three tokens, because this encoding's vocabulary was built before the word became common. Now try this:

enc.encode("strawberry")
# → [496, 675, 15717]    three tokens: "str", "aw", "berry"

enc.encode(" strawberry")
# → [73700]              one token, once there's a space in front

Why is "strawberry" three tokens on its own but one token after a space? Why does every LLM API charge by the token? And why did models like GPT-4 famously struggle to count the r's in "strawberry"?

All of these have the same answer: tokenisation. It's the invisible first step in every LLM pipeline, it's rarely explained well, and it has real consequences for how you build GenAI systems. Get it right and you'll understand why models behave the way they do, how to manage your context window (Article #01, §06: the maximum number of tokens a model can consider in one request, counting your prompt and its reply), how to forecast API costs, and how to spot failure modes before they bite you in production.

§ 02 · The ConceptWhat Is a Token?

A token is the basic unit of text an LLM reads and writes. It's not a word, and it's not a character. It sits in between: a subword unit that balances the size of the vocabulary against the length of the sequence.

Some rules of thumb for English text:

• 1 token ≈ 4 characters
• 1 token ≈ ¾ of a word, so 100 tokens ≈ 75 words
• The cl100k_base encoding used in most of this article (the one behind the original GPT-4 and GPT-3.5-turbo) has a vocabulary of about 100,000 tokens. OpenAI's newer models, from GPT-4o onward, use o200k_base, with about 200,000. The ideas below apply to both; only the exact numbers shift.

Everything you send to an LLM becomes a sequence of integer IDs from that fixed vocabulary. Every word, punctuation mark, number, emoji, and code symbol is represented as some combination of vocabulary items.

Why not just use words?

It's the natural question, but a word-level vocabulary runs into two serious problems. First, the vocabulary explodes. Large English dictionaries list hundreds of thousands of words, and that's before names, technical jargon, typos, slang, and every other language. Second, there's no good way to handle a word the model has never seen. "ChatGPT" didn't exist in any dictionary before late 2022. A word-level model would have to replace it with a generic [UNKNOWN] marker and lose everything about it.

Why not just use characters?

Characters fix the vocabulary problem; a byte-level vocabulary needs only 256 symbols to represent any text at all. But they create a different problem: sequences get very long. "Hello world" is 11 characters but only two tokens.

That's worse than it sounds. Recall from Article #02 that attention lets every token look at every other token. The number of those token-to-token comparisons grows with the square of the sequence length. A sequence 4–5× longer doesn't cost 4–5× more attention work; it costs 16–25× more. On top of that, the model would have to learn words, syllables, and morphemes (the smallest meaningful chunks of a word, like the "-ing" in "running" or the "un-" in "unhappy") from raw letters, a much harder learning problem.

Subword tokenisation is the sweet spot

Common words become a single token. Rare or long words get split into recognisable pieces. The model sees enough regularity to learn patterns efficiently, and the vocabulary stays a manageable size.

Think of LEGO. A fixed kit of standard bricks can build almost anything, including things nobody has built before. You don't need a custom-moulded piece for every object in existence (the word-level problem), and you don't have to build everything from raw plastic pellets (the character-level problem). Subword tokens are that kit for language.

§ 03 · The AlgorithmByte Pair Encoding: How GPT Tokenises

Byte Pair Encoding (BPE) is the algorithm behind the GPT family, from GPT-2 through GPT-4o, and behind the Llama models. It's the dominant approach, so it's worth understanding properly.

The most important thing to know up front: BPE isn't a set of rules someone wrote. Its vocabulary is learned by scanning a huge amount of text and discovering which chunks naturally recur.

The intuition: merge what appears together

Start with the smallest possible units, individual characters. Ask: "Which two neighbouring units appear together most often across all my text?" Merge that pair into one new unit. Repeat. After enough merges, common words become single units, common endings like -ing, -tion, and -est become their own tokens, and rare words stay as several pieces.

Step by step on a toy corpus

Real tokenisers count how often each word appears, so our toy corpus has counts too: "low" appears 5 times, "lower" 2 times, "newest" 6 times, and "widest" 3 times. (This is the classic example from the 2016 paper that first brought BPE to neural machine translation.) Each word starts as separate characters plus an end-of-word marker, </w>.

That marker matters more than it looks. It acts as a fence at the end of each word, so a merge can never join the last letter of one word to the first letter of the next. GPT-style tokenisers do the same job differently: instead of an end marker, they glue the space onto the front of the following word, which is why you'll see tokens like ' is' in §05.

After just eight merges on a tiny corpus, the ending "-est", the base "low", and the whole word "newest" have all become single units. On real data, with tens of thousands of merges, the same loop produces an entire vocabulary of useful pieces.

Why this matters in practice

• Common English words are one token: the, and, is, Python, function (all confirmed with cl100k_base).
• Less common words take two to four: tokenisation → token + isation.
• Many non-English scripts take more tokens for the same meaning, because the training text was mostly English (§07 has the measurements).
• Random strings, IDs, and hashes are expensive: often close to one token per character.

§ 04 · Other AlgorithmsWordPiece and SentencePiece

BPE isn't the only approach. When you use models from Hugging Face, you'll meet two others, and it's worth knowing enough not to be surprised.

§ 05 · Hands-on CodeExploring Tokenisation with tiktoken

This is the first proper coding section of the series. Install tiktoken, open a Jupyter notebook, and run every cell below. Don't just read the outputs; change the inputs and watch what happens. Every output shown here was produced by running this exact code.

$ pip install tiktoken

The basics: encode, count, decode

import tiktoken

# The encoding behind the original GPT-4 and GPT-3.5-turbo (~100k tokens)
enc = tiktoken.get_encoding("cl100k_base")

text = "Tokenisation is the first step in every LLM pipeline."
tokens = enc.encode(text)

print(tokens)
# [3404, 8082, 374, 279, 1176, 3094, 304, 1475, 445, 11237, 15660, 13]

print(len(tokens))
# 12

# Decoding always gives you the original text back
print(enc.decode(tokens))
# Tokenisation is the first step in every LLM pipeline.

Token-by-token inspection

This is the most useful thing you can do: look at what each integer actually stands for.

for token_id in enc.encode("Tokenisation is the first step"):
    print(f"{token_id:6d}  →  {repr(enc.decode([token_id]))}")

#   3404  →  'Token'
#   8082  →  'isation'
#    374  →  ' is'       ← the space is part of the token
#    279  →  ' the'
#   1176  →  ' first'
#   3094  →  ' step'

Two things stand out. First, "Tokenisation" splits into 'Token' and 'isation'. Second, token 374 is ' is', with the space built in, not just 'is'. Think of a rubber stamp that has a blank margin carved into its left edge. The tokeniser learned that "a space followed by a word" appears so often that it's cheaper to treat the pair as one unit than to spend a separate token on every space. (If you print all 12 tokens from experiment 1, you'll also see "LLM" split into ' L' and 'LM'.)

This matters when you assemble prompts: "is" and " is" are different tokens with different IDs, so the way you join strings together can quietly change your token count.

The strawberry problem, live

print(enc.encode("strawberry"))
# [496, 675, 15717]     "str" + "aw" + "berry"

print(enc.encode(" strawberry"))
# [73700]               the whole word, one ID

The model never sees the individual letters. To count the r's, it has to recall what's spelled inside each token and add up the results, and nothing in next-token training teaches that directly. Newer models do much better at this, largely by spelling the word out step by step first (chain-of-thought, Article #01 §16), but the underlying limitation comes from tokenisation itself.

Three more instructive experiments

o200k = tiktoken.get_encoding("o200k_base")   # the GPT-4o encoding (~200k tokens)

for label, text in [("English", "Hello, how are you?"), ("Hindi", "नमस्ते, आप कैसे हैं?")]:
    print(f"{label:8s} cl100k: {len(enc.encode(text)):2d}   o200k: {len(o200k.encode(text)):2d}")

# English  cl100k:  6   o200k:  6
# Hindi    cl100k: 20   o200k:  9

# The older encoding needs 3.3x more tokens for the Hindi greeting.
# The newer one narrows that to 1.5x. This affects cost AND context usage.

print(enc.encode("token"))    # [5963]
print(enc.encode(" token"))   # [4037]   a different ID!

# "Please summarise:" and " Please summarise:" are different token sequences.

code      = "x = [i**2 for i in range(10)]"
prose     = "x equals the list of i squared for i in range ten"
random_id = "a7f3-4b2c-9d8e-1f0a-3c7b"

for label, text in [("code", code), ("prose", prose), ("random", random_id)]:
    n = len(enc.encode(text))
    print(f"{label:6s}: {n:2d} tokens for {len(text):2d} chars  ({len(text)/n:.1f} chars/token)")

# code  : 13 tokens for 29 chars  (2.2 chars/token)
# prose : 12 tokens for 49 chars  (4.1 chars/token)
# random: 24 tokens for 24 chars  (1.0 chars/token)

# Plain English packs the most characters into each token.
# A random ID costs one token per character: the worst case.

Cost estimation: a production skill

def estimate_cost(text: str, usd_per_million_tokens: float) -> dict:
    """Estimate input cost before calling the API. Pass your model's current price."""
    n_tokens = len(enc.encode(text))
    return {"tokens": n_tokens, "cost_usd": n_tokens / 1_000_000 * usd_per_million_tokens}

print(estimate_cost("Tokenisation is the first step in every LLM pipeline.", usd_per_million_tokens=5.0))
# {'tokens': 12, 'cost_usd': 6e-05}

# For a 25,000-token document at the same rate: 25,000 / 1,000,000 x $5 = $0.125 per call.
# Sent 10,000 times a day, that's $1,250 a day. Worth optimising.

§ 06 · Hands-on CodeExploring Hugging Face Tokenisers

When you move from OpenAI's API to open models like BERT, T5, Llama, or Mistral, you'll use Hugging Face's transformers library. The interface is almost the same as tiktoken. The important difference is that each model family splits text in its own way.

$ pip install transformers sentencepiece

from transformers import AutoTokenizer

# BERT uses WordPiece: "##" marks a piece that continues the previous one
bert_tok = AutoTokenizer.from_pretrained("bert-base-uncased")
print(bert_tok.tokenize("Tokenisation is fascinating"))

# T5 uses SentencePiece: "▁" marks where a space was
t5_tok = AutoTokenizer.from_pretrained("t5-small")
print(t5_tok.tokenize("Tokenisation is fascinating"))

Run it and look for the markers: BERT's pieces that continue a word start with ##, and T5's pieces that start a new word begin with ▁. The exact splits depend on each model's learned vocabulary, which is the whole point: the same sentence gets carved up differently by each family.

Same sentence, different counts

sentence = "The Generative AI revolution is transforming software engineering."

for model_name in ["bert-base-uncased", "gpt2", "t5-small"]:
    tok = AutoTokenizer.from_pretrained(model_name)
    print(f"{model_name:18s} → {len(tok.tokenize(sentence)):3d} tokens")

On plain English like this, the counts land close together. Try it again with a Hindi or Tamil sentence, or a block of code, and the gaps open up quickly. That's the pattern Figure 5 measures in the next section.

§ 07 · Engineering JudgementWhat This Means for GenAI Engineers

Theory only matters if it changes how you build. Here's what token awareness looks like in day-to-day engineering.

1. Context window management

Every LLM has a hard ceiling on the number of tokens it can handle in one call. The limit varies by model and keeps growing as providers ship longer-context versions, so check the documentation for the one you use. The ceiling covers everything: your instructions, your few-shot examples (Article #01, §16), any retrieved documents, the conversation so far, and the model's reply.

Think of packing a suitcase with a strict weight limit. Every item (instructions, examples, documents) has a weight in tokens. Common short words pack efficiently, while rare words, code, and non-English text take up more room. Once you hit the limit, nothing else fits, however important it is. When you build RAG systems (Phase 5), measure each chunk's token count before assembling the prompt, so you know it fits before you try to close the suitcase.

2. API cost forecasting

APIs charge separately for input and output tokens, and prices differ a lot between models. As a rough guide, a 50-page document is somewhere around 25,000–35,000 tokens, depending on how dense the text is. At an example rate of $5 per million input tokens, 25,000 tokens costs about $0.125 per call, and 10,000 calls a day comes to $1,250 a day. Check your provider's current pricing page for real figures; prices are among the fastest-changing numbers in the industry. Concise prompts and sensible chunking are engineering decisions, not just style.

3. Prompt design

Every word in your system prompt costs tokens, on every single call. Long, rambling instructions aren't just inelegant; they cost money and use up space that could hold user data or retrieved documents. Few-shot examples are powerful but token-heavy, so choose them carefully. Sometimes one well-chosen example beats three mediocre ones.

4. Multilingual apps

This one matters a lot for Indian-language products. Tokenisers learn from their training text, and older ones saw mostly English. Figure 5 shows the same sentence measured in four languages.

Two lessons follow. First, if you build for Hindi, Tamil, or other Indian languages, measure token counts in those languages; don't budget from English numbers. Second, the choice of model matters: newer tokenisers with bigger, more multilingual vocabularies can cut your costs for the same text by a factor of three or more. Models trained specifically for Indian languages can do better still.

5. The letter-counting problem

This isn't a bug that one patch will fix. It follows directly from subword tokenisation: the model sees IDs, not letters. Models have improved a lot at it, mainly by spelling words out before counting, but when your application needs reliable character-level work (counting letters, checking spelling patterns, validating formats), do it in ordinary code rather than asking the LLM.

§ 08 · Optional Deep DiveBuild a Tiny BPE Tokeniser from Scratch

This section is for readers who want to fully demystify the algorithm. If the explanation in §03 was enough, feel free to skip it. But if you run this code, something will click that no explanation fully replaces.

import re
from collections import defaultdict

def get_vocab(corpus):
    """Split each word into characters plus an end-of-word marker, and count it."""
    vocab = defaultdict(int)
    for word in corpus:
        vocab[' '.join(word) + ' </w>'] += 1
    return vocab

def get_pairs(vocab):
    """Count every pair of neighbouring symbols, weighted by word frequency."""
    pairs = defaultdict(int)
    for word, freq in vocab.items():
        symbols = word.split()
        for i in range(len(symbols) - 1):
            pairs[(symbols[i], symbols[i + 1])] += freq
    return pairs

def merge_vocab(pair, vocab):
    """Merge every occurrence of the pair, matching whole symbols only."""
    pattern = re.compile(r'(?<!\S)' + re.escape(' '.join(pair)) + r'(?!\S)')
    return {pattern.sub(''.join(pair), word): freq for word, freq in vocab.items()}

# The toy corpus from §03, with word counts
corpus = ['low'] * 5 + ['lower'] * 2 + ['newest'] * 6 + ['widest'] * 3
vocab = get_vocab(corpus)

for i in range(10):
    pairs = get_pairs(vocab)
    if not pairs:
        break
    best = max(pairs, key=pairs.get)
    print(f"Merge {i+1:2d}: {best[0]:5s} + {best[1]:5s} → {''.join(best):9s} (seen {pairs[best]}×)")
    vocab = merge_vocab(best, vocab)

Merge  1: e     + s     → es        (seen 9×)
Merge  2: es    + t     → est       (seen 9×)
Merge  3: est   + </w>  → est</w>   (seen 9×)
Merge  4: l     + o     → lo        (seen 7×)
Merge  5: lo    + w     → low       (seen 7×)
Merge  6: n     + e     → ne        (seen 6×)
Merge  7: ne    + w     → new       (seen 6×)
Merge  8: new   + est</w> → newest</w> (seen 6×)
Merge  9: low   + </w>  → low</w>   (seen 5×)
Merge 10: w     + i     → wi        (seen 3×)

Read through the output and you'll see the algorithm discovering that -est and low are valuable units without being told anything about English grammar. It all comes from frequency. The only real difference between this and the tokeniser inside a GPT model is scale: cl100k_base holds about 100,000 learned tokens, built by running the same kind of loop over an enormous amount of text.

§ 09 · SummaryWhat You Now Know