Field Notes / 04

A compact notebook on modern AI

How machines learn to predict language.

A large language model does not store a tiny person inside a computer. It turns text into numbers, moves information through many learned layers, and estimates which token should come next.

Reading time: 7 min Edition 2026.04 Conceptual overview

Text enters as pieces

Words are not the native language of a model.

Before computation begins, a tokenizer divides text into reusable fragments. A fragment may be a word, part of a word, punctuation, or whitespace. Each fragment receives an integer ID.

The model learn s by predict ing .

Token boundaries are learned from patterns in data. They do not always align with the words a reader sees.

Discrete identity
token_id = 18,742
Learned embedding
[0.18, −0.42, 0.07, …]
What the ID does Points to one entry in the model vocabulary.
What the vector does Places that token inside a continuous space of learned relationships.

Selective information flow

Attention asks what matters right now.

The meaning of a token depends on its neighbors. Self-attention lets each position gather information from other positions, using learned relevance scores rather than a fixed rule.

The robot placed the glass down because it was fragile.
glass high
robot low
placed low

Conceptual illustration only. Real models use many attention heads, and their internal relationships are more distributed.

Q

Query

What information is this position looking for?

K

Key

What kind of information does each position offer?

V

Value

What information should be passed forward if selected?

The transformer block

Understanding emerges through repeated refinement.

A transformer is built from a sequence of similar blocks. Each block combines information across the context, transforms it, and preserves a path for earlier signals to continue forward.

Token embedding Convert vocabulary IDs into dense vectors. Input
Position signal Preserve information about order and distance. Structure
Self-attention Exchange information between relevant positions. Context
Feed-forward network Transform each position through learned nonlinear operations. Compute
Residual stream Carry and combine signals across many layers. Memory
Output probabilities Score every possible next token in the vocabulary. Prediction

Two very different phases

Training changes the model. Inference uses it.

The same neural network behaves differently during construction and use. Training is an optimization process; inference is a sequence of forward predictions through parameters that are usually fixed.

Phase A / Training

Learn from error

The model sees text with the next token hidden, makes a prediction, and measures how far that prediction is from the observed answer. Backpropagation assigns responsibility for the error, and an optimizer adjusts billions of numerical parameters.

predict → compare → adjust

Repeated across large datasets, this process compresses many statistical patterns of language into the model weights.

Phase B / Inference

Generate from context

A prompt is converted to tokens and passed through the network. The model produces a probability distribution over the next token. A decoding method selects one candidate, appends it to the context, and starts the cycle again.

context → probability → token

Temperature and sampling rules influence variation, but they do not add knowledge that was absent from the model or its current context.

What prediction does not guarantee

Fluent language is not the same as reliable truth.

Language models are useful because prediction at scale captures rich structure. The same mechanism also creates important limitations.

  1. 01 Hallucination A plausible continuation can still be unsupported, outdated, or false.
  2. 02 Finite context Only information inside the active context can directly shape a response.
  3. 03 Data inheritance Patterns, gaps, and biases in training data can influence model behavior.
  4. 04 Uneven reasoning Performance can change sharply with phrasing, domain, tools, and task complexity.

Margin conclusion

The core mechanism is simple. The learned system is not.

At the surface, a language model repeats one operation: predict the next token. Underneath, millions or billions of learned interactions transform context into that prediction. Scale turns a small objective into a broadly capable—but still fallible—tool.