AI/LLM Mastery · Part 2 of 20 — a computer can only do maths on numbers, so before a model can touch your words it must turn them into numbers. This is how — and why that one quiet step explains some of an LLM’s strangest habits.
First, your words have to become numbers
In Part 1 we landed on a simple truth: a language model is, at heart, a very well-read autocomplete — it reads some text and guesses what comes next. But we quietly skipped over something important. A neural network, as we said, is just a giant bank of numbers being multiplied together. It cannot read the letter “c.” It cannot read the word “cat.” It can only do arithmetic. So before any of the clever stuff can happen, your words have to become numbers.
That conversion is the whole subject of this article. It sounds like dull plumbing — and in a way it is — but it is plumbing that leaks in fascinating ways. By the end you will understand exactly why a model that can write you a sonnet sometimes cannot count the letters in “strawberry,” and why the same sentence can cost an English speaker and a Hindi speaker very different amounts of money. As always, every new word gets defined the moment it shows up.
Where this happens: the tokenizer
Let us see the whole journey first, then zoom in. When you type a message to an AI, it does not go straight into the “brain.” It passes through a small, unglamorous program first.
Your text goes into a tokenizer — a simple program whose only job is to chop the text into small standard pieces. Each piece is then swapped for a number. Only then does the neural network see anything, and what it sees is that row of numbers — never your actual letters. Think of the tokenizer as the doorway every word must walk through to enter the model’s number-only world. There is no intelligence in this step at all; it is pure, mechanical chopping and looking-up. But how it chops turns out to matter enormously.
Two obvious ideas — and why both fail
How should we chop text into numbered pieces? Two obvious ideas come to mind first, and seeing why both fail tells you exactly why the real answer looks the way it does.
The first idea: one number per letter. Give “a” a number, “b” a number, and so on. The list is tiny and tidy. The problem is that a single letter carries almost no meaning — “c,” “a” and “t” on their own tell the model nothing about a cat. It would have to laboriously reassemble the meaning of every word from raw letters, every single time. Wasteful.
The second idea: one number per whole word. Now “cat” is a single, meaningful unit — much better. But count the words: once you include every plural, tense, name and bit of slang, English alone has millions of forms. That list becomes huge and slow. Worse, the moment someone types a brand-new word (“rizz”), a typo, a product name, or a word in another language, there is simply no number for it. The model hits a blank. We need something in between — bigger than letters, but not as brittle as whole words.
The middle ground: tokens
The answer the whole industry settled on is the middle ground: tokens. A token is a chunk of text that is usually a whole common word, but which breaks rarer words into smaller pieces. The fixed list of every token a model knows is called its vocabulary (typically somewhere around 50,000 to 100,000 entries), and the act of chopping text into tokens is called tokenization.
Watch what happens to different words. Common ones like “I,” “love” and “cats” are frequent enough to each earn their own single token. A rarer word like “tokenization” is not, so it gets built from smaller known pieces — “token” + “ization.” Even odd casing splits up: “ChatGPT” often becomes “Chat” + “G” + “PT.” Because any word can be spelled out of smaller pieces — right down to single characters if needed — there is no such thing as an “unknown word” any more. That single property is what makes tokens beat the whole-word approach.
One detail that trips people up: the leading space is usually part of the token. The tokenizer sees “·cat” (space-cat) as a different token from “cat” at the very start of a sentence. It is a small thing, but it is why token lists look slightly odd when you first see them.
How the pieces are chosen: Byte-Pair Encoding
That raises a fair question: who decides which chunks become tokens? Nobody sits down and writes the list by hand. It is learned from data — the same theme as Part 1 — using a beautifully simple, repeated trick called Byte-Pair Encoding, or BPE.
Here is the whole algorithm in plain words. Start with the most basic tokens imaginable: every individual character. Then scan a giant pile of text and find the pair of neighbouring tokens that appears most often — say the letters “i” and “n,” because “in” turns up everywhere. Glue that pair into a single new token, “in,” and add it to the vocabulary. Now repeat: maybe the most common pair is now “in” + “g,” so they merge into “ing.” Merges stack on top of earlier merges, so the chunks keep growing from letters into syllables into whole common words.
Run that loop tens of thousands of times over real text and the useful chunks fall out automatically: common endings like “ing,” “ed” and “·tion,” and whole frequent words like “the” and “cat.” That learned list of merges is the tokenizer’s vocabulary. The data chose it, not a human — which is exactly why English (which most tokenizers are trained on) packs so neatly, and other languages sometimes do not.
The final step: tokens become numbers
We are nearly there. We have chopped the text into tokens; the last step is to turn each token into its number. This part is trivial: every token in the vocabulary has a fixed position in the list, and that position is its ID. Tokenizing a sentence just means looking up each token and writing down its ID.
# 1. split into tokens (note the leading spaces, and "." is its own token) "The cat sat." → ["The", " cat", " sat", "."] # 2. look each token up in the vocabulary → its ID ["The", " cat", " sat", "."] → [464, 3797, 3332, 13] # THIS row of integers — not your letters — is what the model receives.
And that is the punchline of the whole article: the neural network only ever sees that row of integers. Your carefully chosen words reach it as [464, 3797, 3332, 13] and nothing else. It is also why two things you will hear constantly are measured in tokens, not words: context length (how much text a model can read at once) and the price you pay to use one. A handy rule of thumb: in English, one word is roughly 1.3 tokens.
The payoff: why tokens explain the weirdness
Now the reward for understanding all this. A surprising number of an LLM’s most-mocked failures are not mysteries at all once you remember one fact: it sees tokens, not letters.
Ask “how many r’s are in strawberry?” and a powerful model may confidently get it wrong. Why? It never sees s-t-r-a-w-b-e-r-r-y. It sees two tokens, roughly “straw” and “berry.” Counting letters means peering inside tokens, which is exactly the view it does not naturally have — so it guesses. The same explains why spelling a word backwards is oddly hard for it: the letters are bundled inside chunks. It reads the way you read a familiar word — as a whole shape — not letter by letter.
Tokens also explain the fairness gap. Because most tokenizers are trained mainly on English text, English compresses into few tokens, while many other languages and scripts get chopped into far more tokens for the same meaning — meaning users of those languages pay more and wait longer for the same answer. And long numbers split into untidy chunks rather than clean digits, which is one more reason exact arithmetic is shaky (we are not done with that story — tools in Part 18 hand the model a real calculator).
Where this leaves us
So the picture from Part 1 just got one layer sharper. Your text is chopped into tokens by a tokenizer whose vocabulary was learned from data with BPE, and each token is swapped for an integer ID. The model lives entirely in that world of numbers.
But notice what these numbers are not: they are not meaning. The ID 3797 for “·cat” is just a position in a list — it could have been 5000 or 42. The number says nothing about cats being furry, or being closer to “dog” than to “car.” A bare lookup ID is a name tag, not an understanding. So the very next question is: how do we turn these meaningless ID numbers into numbers that actually capture meaning — so that “cat” and “dog” end up close together and “cat” and “Tuesday” far apart? That is embeddings, and it is where Part 3 begins.