How large language models work inside
Follow text through tokenization, vectors, causal attention, transformer blocks, training, and the repeated next token generation loop.
The whole story in 6 lines
An LLM repeatedly turns token context into probability distributions using learned transformer weights, then feeds each chosen token...
- A tokenizer maps raw text into reusable subword pieces from a fixed vocabulary.
- Token and position vectors add together so content and order enter the model in one stream.
- Causal attention mixes earlier information while giving every future position exactly zero weight.
- Residual paths preserve the stream while attention and an MLP add learned refinements.
- Training compares every next token prediction with the known continuation and sends error back into the weights.
- Generation samples one token, appends it to context, and repeats the same prediction loop.
Setup
A chat response can feel instant and fluent, but the model builds it from repeated numerical predictions. We will follow one short sentence through that machinery.
A token is a reusable text piece, while a vector is a row of numbers that carries information about that piece.
Attention decides which earlier positions can contribute. A logit is the raw score produced for one possible next token.
Our journey now connects six jobs: split text, build vectors, mix context, refine the stream, learn from errors, and generate repeatedly.
Keep those four terms nearby as the same token pieces and vectors return across stages. Let us begin with how raw text becomes model input.
Text Becomes Tokens
The model begins with a character string, not with ready made words or meanings. This stage follows that string into the discrete pieces the network can receive.
A subword tokenizer matches reusable pieces from a fixed vocabulary. Common fragments may stay together, while a less common word can split across several pieces.
The word curious has crossed two vocabulary boundaries in our illustrative fixture. What must replace those text pieces before matrix operations can begin?
Every shard now lands in one numbered vocabulary slot. The sequence length comes from the tokenizer, while the actual IDs come from the fixed vocabulary it learned earlier.
Tokenization preserves text as an ordered ID sequence, but an ID alone carries no useful geometry. Those token boundaries stay available as the sequence moves through every later layer. Their order stays fixed throughout that journey. Next, each ID becomes a vector and gains a signal for position.
Meaning Meets Position
We now have ordered vocabulary IDs from tokenization. This stage asks how one small integer becomes a numerical state that can carry both content and location.
Each ID selects one row from a learned embedding table. Tokens used in similar contexts can develop related patterns across these learned vector dimensions.
The same token vector could appear at any location, so content alone cannot distinguish subject order from object order. What signal must enter before attention compares positions?
The semantic strip and position strip add cell by cell. Their combined vector now carries token content and sequence location through the same fixed width stream. This sum does not attach a human definition to the token. It creates a learned state whose coordinates matter only through later computations.
A position aware vector is still only one local state. Next, causal attention will compare these states and mix useful information from earlier tokens.
Causal Attention
★ If you remember one thing · Causal masking removes every future path before attention mixes information.
The vectors now know content and position, but each one is still isolated. Attention lets every query compare with keys and gather a weighted mixture of value vectors.
A query and key dot product produces one compatibility score. Softmax then turns each completed row into weights that sum to one before values are mixed.
During next token training, a position must not copy information from text that comes later. Which part of the score matrix must disappear before softmax?
The contrast is now complete. The unrestricted matrix keeps every cell, while the causal matrix crosses out its future triangle and gives those paths exactly zero weight. Each allowed weight now tells the model how strongly one earlier value should influence the current evolving token state.
Each allowed row now mixes only information available at that position. Next, the transformer block will add that attention result back into the continuing vector stream.
Inside One Transformer Block
Causal attention has produced a context update for every position. One transformer block must fold that update into the stream without discarding the state that arrived.
Layer normalization prepares the stream, then attention computes a context shaped delta. The original stream also travels around that operation on a residual path.
The first addition joins original state and attention delta. A second normalization then prepares each position for an independent multilayer perceptron, often called the MLP.
The MLP expands and transforms each position before projecting it back to the stream width. What should happen to the state that entered this second branch?
The second addition preserves the incoming state and adds the MLP refinement. Attention mixes information across positions, while the MLP transforms each position through learned nonlinear features.
One block makes two refinements without replacing the stream. Deep models repeat this structure many times. Next, we will see how training teaches all those weights.
Learning From the Next Token
The transformer block contains learned weights, but architecture alone does not choose useful values. Training turns ordinary text into many next token exercises for those weights.
The training target is the same sequence shifted one position. After The, the target is the first curious piece. After that piece, the target is its continuation.
Every position produces logits over possible vocabulary tokens. Once softmax assigns probability to the true continuation, what signal tells the model how wrong it was?
The computed losses now flow backward through every operation. An optimizer uses those gradients to update shared weights, so future passes can assign more probability to the actual continuation. Those shared weights affect many positions, so each update combines evidence from many prediction examples.
Training repeats this prediction and update process across vast text collections. At runtime the updates stop. Next, fixed weights will generate by choosing and appending one token at a time.
One Token, Then Another
Training has finished and the weights are fixed. Generation begins with the current token context and runs one forward pass through the trained transformer stack.
The final hidden state projects into one logit for every vocabulary entry. Softmax converts those raw scores into a probability distribution for the next position.
Temperature rescales the logits before softmax. If Temperature rises, how will the probability bars change while their ordering stays the same?
One token is selected from the computed distribution. The bars show uncertainty before selection, while the green shard records the concrete continuation chosen for this run.
Switch Temperature through every setting and compare how strongly the longest bar dominates. Your goal is to make one candidate clearly outweigh the others.
The selected shard appends to context and becomes input for the next pass. Repeating this same loop builds a response one token at a time. Now let us connect the entire system.
The Whole LLM Loop
We started with tokenization, which turned raw characters into an ordered sequence of reusable vocabulary pieces and IDs.
Then embeddings and position signals turned those IDs into vectors that carry both content and sequence order.
Causal attention compared positions, removed every future path, and mixed only information available at each prediction point.
Transformer blocks preserved the stream through residual paths while attention and MLP branches added learned refinements.
Training shifted real text into next token targets, measured prediction loss, and pushed gradients backward into shared weights.
Generation reused those fixed weights, sampled one token from computed probabilities, and appended it to the continuing context.
The complete system now meets at one repeated task: turn the available token context into a probability distribution for what comes next.
The whole story in 6 lines
An LLM repeatedly turns token context into probability distributions using learned transformer weights, then feeds each chosen token...
- A tokenizer maps raw text into reusable subword pieces from a fixed vocabulary.
- Token and position vectors add together so content and order enter the model in one stream.
- Causal attention mixes earlier information while giving every future position exactly zero weight.
- Residual paths preserve the stream while attention and an MLP add learned refinements.
- Training compares every next token prediction with the known continuation and sends error back into the weights.
- Generation samples one token, appends it to context, and repeats the same prediction loop.