How LLM inference works from prompt to streamed tokens
Trace prompt formatting, tokenization, batching, prefill, KV cache, autoregressive decode, sampling, and streaming through one LLM request.
The whole story in 6 lines
LLM inference schedules prefill and decode to turn one token sequence into sampled tokens while preserving reusable state.
- Chat templates create one model specific token sequence from roles, content, and control markers.
- Iteration level scheduling replaces finished requests quickly, which keeps expensive accelerator slots useful.
- Prefill processes the prompt in parallel and creates the attention state needed for the first prediction.
- KV caching cuts repeat work. Prefix sharing varies by engine.
- Autoregressive decode predicts one token at a time, and sampling turns logits into a controllable choice.
- Streaming can improve time to first displayed text even when end to end generation time stays unchanged.
The hidden lifecycle
You press Send on a six word question, but the first answer token still takes time to appear. The delay hides several different jobs across software and accelerator memory.
A token is a model symbol rather than a full word. Prefill reads the prompt, decode adds output tokens, and a KV cache remembers attention state.
Our request changes shape as it moves from message roles into token rows, batch lanes, attention cells, cache pages, and an output stream. Every picture follows the same prompt.
The lifecycle has two broad compute phases surrounded by ordinary serving work. Now let us start with the exact sequence that reaches the model.
Prompt to tokens
We begin with two application messages, one system instruction and one user question. They look separate in the interface, but a causal language model continues one sequence.
A chat template inserts model specific control markers around each role. Hugging Face documents that different chat models may expect different markers even when their base architecture is related.
The tokenizer now cuts that formatted text into subword pieces and maps each piece to a vocabulary entry. What does the model finally receive?
The two message cards collapse into one fifteen piece teaching sequence with role markers preserved. Our pieces are illustrative because the exact split depends on the selected tokenizer and chat template.
Formatting is part of model compatibility because control tokens carry the role boundaries learned during chat training. Those boundaries also keep instructions attached to their intended speakers. Next, the serving system must decide when this token row can use the accelerator.
Batch and schedule
The token row joins requests A, B, and C at the serving queue. A scheduler groups work so accelerator kernels can process several sequences together instead of launching each request alone.
Requests need different numbers of output tokens, so they do not finish together. Orca describes scheduling at iteration granularity, which lets a server reconsider membership after each decode step.
Request B finishes while A still needs two steps and C waits. Under a fixed batch, what happens to B’s accelerator lane?
The contrast now appears across the same five iterations. Fixed membership wastes open cells, while continuous batching moves request C into the freed lane at the next scheduling boundary.
Switch the Scheduler mode control through both policies. Compare the downstream lane outcome and finish after you have exposed both a waiting slot and a reused slot.
Batching raises total hardware utilization, but queue delay and prefill admission still affect individual latency. Next, request A enters the heavier prompt processing phase.
Prefill the prompt
The scheduler admitted request A, so prefill can process its complete prompt. Unlike decode, prompt positions are already known and can move through the model together.
Inside each causal attention layer, a prompt position may use itself and earlier positions but not later ones. That rule forms a lower triangle rather than a fully connected square.
The last prompt position can attend across the entire known prefix. Which region must remain blocked to prevent information from arriving from the future?
The lower triangle fills across all fifteen prompt positions while the future cells stay crossed out. Prefill also writes key and value state for every known position and produces first token logits.
Prefill cost grows with prompt work, so long contexts can dominate time to first token. Every admitted prompt also competes for accelerator capacity. The useful output is not text yet. Next, we inspect the memory that prefill leaves behind.
Build and reuse KV
★ If you remember one thing · A KV cache keeps prior attention state, so decode appends one new slot instead of rebuilding the whole prefix.
Prefill leaves one key and one value state per prompt position in each attention layer. We draw those states as paged cells because serving systems must manage their memory as sequences grow.
Without saved state, every decode step would rebuild keys and values for the growing prefix. A KV cache keeps the old columns, then adds state for only the newly accepted token.
Both lanes now need attention over the same sixteen known positions. Which lane should add only one new slot instead of rebuilding all earlier slots?
The recompute lane redraws the full prefix while the cache lane preserves four filled pages and appends one green slot. Memory replaces a growing amount of repeated computation.
Switch the Prefix cache control through both outcomes. Compare the downstream compute marker and finish after you have exposed a cold full prefill and a reused shared prefix.
Prefix reuse is a provider and server feature, so matching rules and eviction policies vary. The ordinary per-request KV cache still powers the next phase, where output grows one token at a time.
Decode and sample
Now that cached state exists, decode feeds the latest token through the model and reads attention against prior keys and values. The output is a logit for every vocabulary candidate.
Temperature rescales those logits before softmax turns them into probabilities. Which setting will make the leading token more dominant?
One candidate is selected and appended to the sequence, then its key and value state enters the cache. The next decode iteration depends on that accepted result, which enforces the token by token loop.
Switch the Temperature control through both settings. Compare the downstream probability bars and finish after you have exposed a dominant leader and a wider choice distribution.
Sampling changes the choice distribution, but each accepted token still advances the same sequential decode loop. This dependency also limits how aggressively one sequence can use parallel decoding. Next, we separate model completion time from when the user first sees useful text.
Stream and measure
The decode loop has chosen four illustrative output pieces. A delivery layer can hold them until completion or forward partial events while later tokens are still being generated.
Time to first token includes queueing, prefill, the first decode step, and transport. Completion latency includes every remaining decode step, while throughput counts useful tokens over time across requests.
Both modes finish after the same illustrative compute timeline. Which mode can paint the first word before the fourth token exists?
The streaming lane paints its first piece near one hundred illustrative milliseconds, while the buffered lane stays empty until completion. Both lanes still finish near one hundred eighty four milliseconds.
Switch the Delivery mode control through both paths. Compare the downstream first paint marker and finish after you have exposed early output and the full response wait.
Streaming improves perceived responsiveness, but batching and cache memory determine how much total work the server can sustain. Now let us step back and see the whole picture together.
The whole lifecycle
We started with chat messages and learned that a model specific template plus tokenizer creates one ordered symbol sequence. That sequence is the real input to inference.
Then we placed requests into changing batch lanes. Iteration level scheduling can replace finished work quickly, which keeps accelerator capacity useful across uneven output lengths.
Prefill processed the known prompt positions together and filled a causal attention triangle. Its lasting output was the first prediction state plus keys and values for every prompt position.
The KV cache preserved those states, so later decode steps appended new entries instead of rebuilding the prefix. Optional cross-request prefix reuse remained a server-specific optimization.
Decode used cached context to produce logits, then sampling chose one token and extended the sequence. That new token became a required input for the following iteration.
Finally, streaming exposed partial text before generation completed. We separated first token latency from completion latency and total serving throughput, because each metric answers a different question.
The complete lifecycle is now connected around one request. The worked timings and token pieces were illustrative, while the mechanisms came from primary papers and official serving documentation.
The whole story in 6 lines
LLM inference schedules prefill and decode to turn one token sequence into sampled tokens while preserving reusable state.
- Chat templates create one model specific token sequence from roles, content, and control markers.
- Iteration level scheduling replaces finished requests quickly, which keeps expensive accelerator slots useful.
- Prefill processes the prompt in parallel and creates the attention state needed for the first prediction.
- KV caching cuts repeat work. Prefix sharing varies by engine.
- Autoregressive decode predicts one token at a time, and sampling turns logits into a controllable choice.
- Streaming can improve time to first displayed text even when end to end generation time stays unchanged.