A tiny tokenizer from scratch

Tokenization is one of the first steps in NLP. Before a model can understand text, we need to turn raw strings into a format it can process. In the simplest setup, we split text into words and punctuation marks, then map each item to a unique integer.

That idea feels small, but it is the foundation of almost everything that follows in language modeling.

Why tokenization matters

Take a sentence like:

“Hello, world. This is a test.”

A tokenizer might convert it into something like:

["Hello", ",", "world", ".", "This", "is", "a", "test", "."]

Now each token can be assigned an ID. For example:

{"Hello": 0, "world": 1, "This": 2, "is": 3, "a": 4, "test": 5}

This is the basic idea behind vocabulary building: a model sees numbers, not raw text.

Splitting the text

A simple tokenizer often uses regular expressions to separate whitespace and punctuation. This helps keep words and punctuation distinct, which is useful for later language modeling tasks.

import re

text = "Hello, world. Is this-- a test?"
tokens = re.split(r'([,.:;?_!"()\']|--|\s)', text)
tokens = [item.strip() for item in tokens if item.strip()]
print(tokens)

This gives us a clean list of meaningful pieces from the text.

Building a vocabulary

Once we have tokenized a corpus, we can collect the unique tokens and create a mapping from token to integer ID:

all_words = sorted(set(tokens))
vocab = {token: i for i, token in enumerate(all_words)}

Now every token has a stable numeric representation.

Handling unknown words

Real text contains words the model has never seen before. A practical tokenizer adds a special token such as <|unk|> for unknown words:

word = token if token in vocab else "<|unk|>"

This makes the pipeline more robust.

Reference

This post follows the same idea as the notebook in GitHub: embeddings.ipynb, which is based on the book Natural Language Processing from Scratch.

Ending Notes

The core flow is very simple:

1. split text into tokens
2. collect unique tokens into a vocabulary
3. map each token to an integer ID
4. handle unknown tokens safely

That is the first real bridge between human language and machine learning.

It is a tiny step, but it is one of the most important ones in NLP.