Byte Pair Encoding (BPE)

Byte Pair Encoding (BPE) is a subword tokenization method that sits between word-level and character-level approaches. Instead of treating every word as a single unit, it learns smaller reusable pieces from the text itself.

That is why BPE is such a useful idea in modern NLP: it gives us a compact vocabulary while still handling rare words and variations more gracefully than pure word tokenization.

Getting started with a tokenizer

Lets see the GPT-2 tokenizer from tiktoken:

from importlib.metadata import version
import tiktoken
print("tiktoken version:", version("tiktoken"))

Then it creates the tokenizer:

tokenizer = tiktoken.get_encoding("gpt2")

This is the basic setup used in many modern tokenization pipelines.

Encoding and decoding text

The first practical example encodes a short sentence and then decodes it back:

text = (
    "Hello, do you like tea? <|endoftext|> In the sunlit terraces"
     "of someunknownPlace."
)
integers = tokenizer.encode(text, allowed_special={"<|endoftext|>"})
print(integers)

strings = tokenizer.decode(integers)
print(strings)

This shows the core idea clearly: raw text goes in, integer IDs come out, and the tokenizer can reconstruct the original string afterward.

An unusual word

The notebook also tests a shorter, more unusual string:

text = (
    "Akwirwier"
)
integers = tokenizer.encode(text, allowed_special={"<|endoftext|>"})
print(integers)

strings = tokenizer.decode(integers)
print(strings)

This is useful because it shows that tokenization is not just splitting on whitespace or words. The model is operating on learned token pieces, which is exactly why BPE is so important.

Working with a real text file

The next step is to encode a larger corpus:

with open("the-verdict.txt", "r", encoding="utf-8") as f:
    raw_text = f.read()

enc_text = tokenizer.encode(raw_text)
print(len(enc_text))

Now the text has been turned into a sequence of token IDs. This is the same kind of representation that language models work with during training and generation.

Building context and target pairs

The notebook then creates a small sample and defines context-target examples:

enc_sample = enc_text[50:]
context_size = 4         #1
x = enc_sample[:context_size]
y = enc_sample[1:context_size+1]
print(f"x: {x}")
print(f"y:      {y}")

This is a very important idea in language modeling: the model sees a context window and tries to predict the next token.

We can also inspect those pairs in a more human-readable way:

for i in range(1, context_size+1):
    context = enc_sample[:i]
    desired = enc_sample[i]
    print(context, "---->", desired)

for i in range(1, context_size+1):
    context = enc_sample[:i]
    desired = enc_sample[i]
    print(tokenizer.decode(context), "---->", tokenizer.decode([desired]))

This gives us a concrete view of how training examples are built. Each token is treated as part of a sequence, and the model learns to predict what comes next.

Why this matters

BPE is powerful because it makes tokenization more flexible than plain word splitting. It keeps common patterns together while allowing rare or unfamiliar text to be represented using smaller pieces.

That is why it shows up in systems like GPT-style models: it balances expressiveness, efficiency, and vocabulary size.

In other words, before a model learns language, it first learns how to break language into useful pieces.

Reference

This post follows the same idea as the notebook in GitHub: bpe.ipynb, which is based on the book Natural Language Processing from Scratch.