Language as numbers

Before a model can 'read', words must become numbers. Understanding tokens and embeddings explains a surprising amount of LLM behaviour — from pricing to odd mistakes.

4.1 Computers need numbers

Neural networks only work with numbers. So before a language model can process text, the text has to be turned into numbers. This happens in two steps: tokenisation and embedding.

4.2 Tokens

A token is a chunk of text — often a whole short word, sometimes part of a longer word, a space or a punctuation mark. For example, a word like unbelievable might be split into pieces such as un + believ + able. As a rough rule of thumb for English, a token averages about three-quarters of a word.

Tokens matter in practice because:

• AI services usually charge by the token — for both input and output.
• Models have a limit on how many tokens they can handle at once (the context window).
• Tokenisation explains some odd failures. A model that sees strawberry as a couple of chunks, not ten letters, can struggle to count the letters in it.
• Languages less common in training data often need more tokens for the same meaning, which can make them slower and more expensive to process.

4.3 Embeddings: meaning as coordinates

Each token is turned into an embedding: a long list of numbers that acts like coordinates in a "meaning space" with hundreds or thousands of dimensions. Training arranges this space so that related meanings end up close together — doctor near nurse and hospital; Paris near London.

Embeddings are also used directly in many AI applications. Search that finds documents by meaning rather than exact keywords — the basis of retrieval-augmented generation (RAG) — compares embeddings.

4.4 The core task: predict the next token

At heart, a large language model does one thing: given a sequence of tokens, it predicts a probability for every possible next token. It picks one, adds it to the sequence, and repeats — one token at a time — until it decides to stop.

It sounds simple, but predicting the next token well across trillions of words of text requires learning grammar, facts, styles, reasoning patterns and much more.

4.5 Sampling and "temperature"

The model doesn't have to choose the single most likely token. Sampling settings control how adventurous it is. A low temperature makes it pick likely tokens (more predictable); a higher temperature allows less likely ones (more varied and creative, but more prone to errors). This is why you can ask the same question twice and get different answers.

4.6 The context window

The context window is everything the model can "see" at once: system instructions, your conversation so far, any documents and its own reply. Current models have windows of hundreds of thousands of tokens or more — enough for long reports or codebases. But the model has no memory beyond it unless the application adds one, and very long contexts can still cause it to miss details.

import random
from collections import defaultdict

text = ("the cat sat on the mat . the dog sat on the rug . "
        "the cat chased the dog . the dog chased the ball .")
words = text.split()

# "Training": count which word follows which
follows = defaultdict(list)
for current, nxt in zip(words, words[1:]):
    follows[current].append(nxt)

# "Inference": generate by repeatedly sampling a likely next word
word = "the"
output = [word]
for _ in range(10):
    word = random.choice(follows[word])
    output.append(word)
print(" ".join(output))

Key takeaways

• Text is split into tokens — roughly three-quarters of a word each in English.
• Embeddings place related meanings close together — and can carry bias.
• LLMs generate one token at a time by predicting probabilities.
• Temperature trades predictability for variety.
• The context window is all the model can see at once.