How to design a ChatGPT-scale GPU serving platform
Design an LLM GPU serving platform across streaming SLOs, model memory, parallelism, KV cache, scheduling, resilience, autoscaling, and cost.
Lesson summary
A production LLM platform turns a streaming contract into safe GPU admission, phase-aware scheduling, resilient capacity, and measurable...
- Streaming sends ordered partial events under one response identity and ends exactly once.
- Time to first token and inter-token delay protect different parts of the user experience.
- Tail lengths and burst distributions drive capacity more safely than one average request.
- A stable control snapshot keeps the token data path independent from slow fleet changes.
- Authentication, quotas, and safety reject unsuitable work before it consumes GPU capacity.
- Routing checks compatibility first, then balances load and useful cache affinity.
- Compute, HBM capacity, memory bandwidth, and interconnects constrain inference separately.
- Weight precision changes device count and the HBM left for runtime state.
- A loaded model is not ready until kernels, communication, and representative shapes are warm.
- The runtime receives an ordered token sequence plus explicit generation limits.
- Prefill processes known prompt positions in parallel and creates the initial KV state.
- Decode accepts one token per iteration because each choice becomes the next input.
- Prefill can fill compute while narrow decode often waits on memory movement.
- Each token stores paired key and value state across every layer and KV head.
- KV memory grows linearly with tokens and turns context limits into HBM obligations.
- Weight precision and runtime reserve set an HBM-only ceiling on complete KV reservations per GPU.
- Tensor parallelism shards layers but introduces collectives inside model execution.
- Pipeline parallelism assigns layer ranges and trades communication for fill and drain bubbles.
- Data parallel replicas scale independent request throughput while duplicating weights and caches.
- Topology-aware placement keeps frequent collectives on the fastest local links.
- Safe admission reserves future KV growth before a request enters the active set.
- Iteration-level scheduling replaces finished sequences instead of leaving batch holes.
- Chunked prefill bounds decoder stalls while preserving the total prompt work.
- Paged KV allocation reuses scattered blocks and avoids contiguous maximum reservations.
- Prefix caching reuses only exact leading blocks and computes every unmatched suffix.
- Separate prefill and decode pools trade lower interference for KV transfer and complexity.
- Autoscaling must predict ready capacity because model loading and warm-up take time.
- A partial stream ends explicitly, while a retry starts under a new response identity.
- Correlated traces connect user SLOs to GPU pressure, failures, utilization, and cost.
What makes a streamed response complete?
One terminal event closes the ordered event sequence for that response identity. Text deltas alone do not make it final.
Why track first-token and inter-token latency separately?
The first token includes queue and prefill. Later token gaps mainly expose decode scheduling and delivery.
Why is average request size insufficient for capacity planning?
Averages hide long prompts, long outputs, and synchronized bursts. Those tails create the hardest memory and queue pressure.
Why can the data plane use a stable control snapshot?
It avoids a central policy lookup on every token path and can keep serving while new configuration updates pause.
Why should policy checks happen before the GPU queue?
They reject invalid, unfair, or unsafe work before it occupies scarce accelerator memory and scheduling slots.
What must the router check before replica load?
It must first ensure model and capability compatibility. Load and cache affinity matter only among valid warm replicas.
Which GPU resources can constrain inference independently?
Tensor compute, HBM capacity, HBM bandwidth, and interconnect bandwidth can each become the limiting resource.
How does weight precision affect serving capacity?
Fewer bytes per parameter can reduce the number of devices and leave more HBM for KV state and runtime buffers.
Why should readiness wait after weights reach HBM?
Kernels, workspaces, communication groups, and representative execution shapes may still need preparation before predictable service.
What does admission receive after prompt preparation?
It receives one ordered token sequence plus model, tenant, streaming, and output-limit metadata.
Why can prefill process many positions together?
Every prompt token is already known, so dense matrix operations can process positions in parallel while causal masking blocks future access.
Why does ordinary decode accept one token per iteration?
Each accepted token changes the sequence and KV state used to predict the following token.
Why is narrow decode often memory-bound?
It performs relatively little arithmetic for each byte of weights and cache moved from HBM.
What repeats for each cached token?
A key and a value vector are stored for every layer and KV head at the configured element width.
Which factor makes KV memory grow during generation?
The token-count factor grows with prompt and output length while the model-shape factors stay fixed.
Why does weight precision change the HBM-only KV reservation ceiling?
Weights and KV reservations share finite HBM, so smaller weight formats leave room for more complete sequence reservations. Compute, scheduling, activations, and fragmentation can make actual concurrency lower.
What cost does tensor parallelism add?
Workers must exchange or reduce partial results during model execution, often inside many layers.
What creates a pipeline bubble?
Later depth stages wait while early microbatches fill the pipeline, and early stages wait again during drain.
Why can data-parallel replicas serve independently?
Each replica has a complete model execution group and its own scheduler and KV pool.
Why keep a tensor group inside one fast-link domain?
Its frequent collectives avoid slower cross-node hops and reduce contention with other transfers.
What memory should admission reserve?
It should reserve the request’s allowed future KV growth plus required weights and runtime headroom.
What happens at an iteration boundary?
Finished sequences leave and eligible waiting requests can enter the next active batch.
How does chunked prefill help active decoders?
It breaks one long blocking interval into bounded pieces and schedules decode work between them.
Why can paged KV use scattered physical slots?
A logical block table maps sequence blocks to physical slots, so each request no longer needs one contiguous region.
What part of a request can a prefix hit reuse?
Only exact leading token blocks with compatible model context can reuse prior KV state.
What new cost appears after separating prefill and decode?
Completed KV state must move between pools, and the extra placement and operating complexity must be managed.
Why is GPU utilization alone a late scaling signal?
Queues can grow before utilization triggers, and a new replica still needs loading and warm-up time.
Why not silently replay a partially streamed response?
The client has already received tokens, so replay can duplicate or diverge from the response it knows.
What should one request trace connect?
It should connect phase latency and terminal outcome with HBM, batch occupancy, collective delay, warm capacity, and cost.
Stage 1 of 31
The hidden platform
You press Send as traffic spikes. Text should begin quickly and keep arriving smoothly, even though your request is about to compete with thousands of others for finite GPU memory.
Four terms will let us follow what happens. TTFT is the wait for the first token. Prefill reads the prompt, decode chooses later tokens, and the KV cache preserves attention state between them.
A simple picture would send the request to an idle GPU. We are going to test why that picture is incomplete, from the stream contract through GPU execution, scheduling, memory, and fleet recovery.
The mystery is not merely how a GPU generates text. It is how one response promise survives every queue, memory reservation, batch change, and failure underneath it. We begin with the promise itself.
Stage 2 of 31
API and stream contract
The answer on your screen is not one finished message arriving all at once. It begins as request `req_gpu_design_42`, asking for one model and at most 512 output tokens.
Because streaming is on, the server opens that response before the model has finished. A creation event gives the conversation a single identity, then text deltas extend it.
That early delivery is useful, but it creates an obligation. Once a client has received a delta, the platform cannot later pretend those bytes belonged to some other response.
So the stream needs one event that means no more deltas belong to this response. Which event should make it final?
Which event should make the response final?
The tape can now overlap generation with delivery without making completion vague. Ordered deltas accumulate under one identity, and the terminal event closes that identity exactly once.
We have our first invariant: internal work may move or batch, but the client sees one ordered stream with one ending. What the client feels inside that stream, however, is really two kinds of waiting.
Stage 3 of 31
Set the service objectives
Before the first delta, our request can wait in a queue and run prefill. Time to first token, or TTFT, includes all of that silence.
After text begins, a different delay matters: the gap between successive tokens. A fast opening followed by long pauses still feels slow, so one latency target cannot protect both experiences.
Suppose a burst has already spent most of the waiting allowance before prefill begins. Which policy preserves more of the 700 millisecond budget for the first token?
Which policy protects the first token during a burst?
First-token priority does not create extra time. It makes the queue surrender sooner, leaving 420 milliseconds for prefill and a larger reserve inside the same illustrative total.
Switch the Latency policy through both choices. Compare the queue deadline and prefill allowance while the total 700 millisecond budget remains unchanged.
An SLO is therefore a spending limit for the mechanisms we have not designed yet. To know whether a queue can respect it, we need more than an average request rate.
Stage 4 of 31
Model the workload
At 120 requests per second, the fleet can look comfortably sized. That single number hides whether those requests are short chats, long documents, or different models entirely.
Our worked traffic has a 640-token median prompt but a 4096-token p95. Long prompts spend more TTFT in prefill and allocate much more attention memory at admission.
Output length creates a different kind of pressure. A long answer keeps its decode slot and memory alive after a short answer has already left the GPU.
Now arrivals jump from 120 to 360 requests per second together. Which capacity view exposes that queue risk before the average catches up?
Which view exposes burst risk?
The distribution keeps the three-times burst, the long prompt tail, and the model mix visible at once. Those requests will compete in different phases even though they share one entrance.
This is the load our design must survive, not the tidy average. It also reveals two cadences: traffic changes every moment, while model placement and policy change far more slowly.
Stage 5 of 31
Separate control and data planes
It would be tempting to let every token ask a central service where its model belongs. Under a burst, that would place slow fleet decisions directly inside the latency path.
So the control plane works at the slower cadence. It publishes model versions, placement, quotas, and rollout policy as a versioned snapshot.
Workers read that snapshot, then the data plane handles requests, schedules GPU work, and emits stream events without another policy lookup on each decode iteration.
The separation is also a failure boundary. Serving does not need a fresh placement decision for a model replica that is already warm and known-good.
When the control link breaks, rollouts stop but the wide data lane keeps moving. Existing replicas continue from the last valid snapshot instead of turning a policy outage into a serving outage.
The fast path is now independent, but it is still expensive. Before our blue request earns a place in a GPU queue, the platform should reject work that never belonged there.
Stage 6 of 31
Authenticate, limit, and protect
Authentication is the first cheap test. It turns an anonymous packet into accountable work, which gives later limits a tenant and session to protect.
Quota must count both requests and tokens. A one-line chat and a 4096-token prompt are each one request, but they do not ask the fleet for comparable work.
Safety policy can inspect input before inference and output while it streams. Public OpenAI guidance supports these checks, though the private production path is not published.
These gates are not merely paperwork around the model. Every early rejection preserves queue time, HBM, and decode slots for work that can actually be served.
The two rejected cards leave before the GPU queue, while only our accepted request advances. Cheap policy work has just protected the most scarce part of the system.
Acceptance still does not tell us which GPU should run the request. The model name narrows the answer first, then warmth and reusable state decide among compatible replicas.
Stage 7 of 31
Route to a warm replica
Our request asks for the illustrative deep model. A lightly loaded fast-model replica is still useless because it does not hold compatible weights or execution features.
The router therefore removes incompatible pools before comparing load. What remains is a set of warm replicas that can all honor the same request contract.
Among those replicas, locality can outweigh a small load difference. A matching prefix may remove thousands of prompt tokens from prefill, while session affinity can preserve live KV state.
That makes routing a constrained optimization rather than round robin. Compatibility is mandatory, warmth avoids a cold start, and cached state can save work already performed.
Deep replica two wins because it satisfies all three conditions and already holds our prefix. The other pools stay available for requests whose model contracts differ.
We have found the worker, but “warm GPU” is still an abstraction. To see why capacity is difficult, we need to open that worker and separate the resources hidden inside it.
Stage 8 of 31
Read the GPU as a machine
Inside the chosen worker, tensor cores perform the dense matrix operations that dominate model execution. Many compute tiles can run at once when a phase exposes enough parallel work.
Beside them sits 80 GiB of HBM. It must hold the model weights, runtime buffers, and every active request’s KV state at the same time.
Capacity is only one memory limit. NVIDIA publishes 3.35 terabytes per second of HBM bandwidth for the H100 SXM, because execution must repeatedly move those stored bytes too.
When a model spans GPUs, NVLink carries partial tensors between them. Its 900 gigabytes per second is fast, but each collective still spends time and depends on placement.
The cutaway now shows why “GPU utilization” is not one resource. Dense work fills compute tiles, while narrow work can leave math idle as weights stream out of HBM.
Compute, HBM capacity, HBM bandwidth, and links can each become the bottleneck. The first capacity claim to test is whether the model weights fit at all.
Stage 9 of 31
Size model weights
The weights are the largest resident object. For our illustrative 70-billion-parameter model, their size is simply parameter count multiplied by stored bytes per parameter.
FP16 uses two bytes per parameter, producing 140 billion bytes before runtime overhead. The ribbon crosses the 80 GiB boundary, so one device cannot hold it.
Lower precision stores the same 70 billion parameters with fewer bytes. Which format first brings the entire weight ribbon inside one reference GPU?
Which format first fits the weights on one 80 GiB GPU?
FP16 requires two devices, FP8 fits one with modest room, and INT4 leaves a much larger empty region. Precision changes placement before a single token is processed.
Try every Weight precision. Compare the device count and HBM headroom, which will later compete with the request’s growing KV state.
Fitting the weights only gives us a loaded process. The first real request would still pay for kernel selection, buffers, communication setup, and cold execution shapes unless readiness waits.
Stage 10 of 31
Load and warm the model
Startup first reads and validates weight shards, then copies each shard into the HBM owned by its worker. At that moment the model is present, but not ready.
The runtime still has to choose kernels, allocate workspaces, and create communication groups. Some execution shapes trigger compilation or autotuning the first time they appear.
If readiness opened here, our request would become the warm-up experiment. Its TTFT would include preparation work that later requests never see.
Representative prompt and decode passes exercise those cold paths before traffic arrives. NVIDIA Triton documents model warmup for exactly this initial-delay problem.
Only after weights, kernels, collectives, and representative shapes are prepared does the gate open. The first user now reaches the same ready path as the users after them.
A warm replica is scarce inventory with a real startup delay. Our request can finally enter it, but the worker does not consume chat bubbles. It consumes one exact token sequence.
Stage 11 of 31
Prepare the prompt
The application still sees separate system and user messages. The model does not. Its template must turn those roles and contents into one ordered text sequence.
The template inserts model-specific control markers, then the tokenizer maps text pieces to identifiers. Exact token IDs vary, so our fixture preserves computed counts instead of inventing pieces.
The request keeps a sidecar too: model, tenant, streaming choice, and maximum output. These bounds tell later gates how much work and memory the sequence may demand.
This distinction matters because the scheduler cannot reserve “a chat.” It can reserve a known prompt length and an allowed future sequence length.
The message cards collapse into our 4096-token tape beside a 512-token output limit. From here on, every memory and scheduling decision follows those same declared bounds.
All prompt positions are now known at once. That lets the first model phase do something decode cannot: process many positions together before any output token exists.
Stage 12 of 31
Run prefill
The 4096-token tape enters prefill as a known whole. Unlike future output, none of these prompt positions has to wait for the model to choose an earlier one.
That lets matrix kernels process many positions together. Causal attention still hides future positions from earlier ones, which is why the allowed cells form a lower triangle.
The triangle is a rule about information, not a rule that prefill must run one token at a time. All known rows can contribute parallel work on the GPU.
At every layer, prefill writes a key and a value for each prompt position. The final position also produces scores from which the first output token can be chosen.
The full prompt wave leaves behind a wall of KV pages. That stored attention state is the durable result of prefill, and the first token can now be selected.
We bought the first token with a large parallel calculation and a large memory output. The next token is different because its input does not exist until the current choice is accepted.
Stage 13 of 31
Run decode
Decode begins with the prompt cache and the newest accepted token. It reads earlier keys and values, then produces a score for every candidate in the vocabulary.
Those candidate scores exist together, but the sequence cannot keep them all. Sampling or another decoding rule chooses one authoritative next token.
Only that winner receives a new key and value. The cache grows by one position, and the accepted token becomes input to the following iteration.
This dependency is the constraint: token two depends on token one, which depends on the prompt. Ordinary decode cannot accept an entire future answer in parallel.
The green token lands on the output tape, gains its KV slot, and curves back as the next input. Decode is not one pass but a repeated acceptance loop.
The same model has now produced two very different workloads: one wide prompt wave and many narrow token loops. That difference changes which part of the GPU holds us back.
Stage 14 of 31
Contrast the hardware stress
Put prefill and decode on identical GPUs and they still do not look like the same job. The important difference is how much arithmetic each phase performs per byte moved.
Prefill combines thousands of known positions into large matrix operations. The loaded weights are reused across substantial arithmetic, so the compute tiles can fill.
Decode may have only a narrow active batch, yet it still revisits the model weights and the growing cache for every accepted token.
With little arithmetic available for each byte moved, which GPU resource often becomes the stronger limit during narrow decode?
Which resource often limits narrow decode?
The equal silhouettes finally diverge. Prefill fills the compute field, while decode drives a broad HBM sweep into one narrow active column. One GPU has become two machines.
Prefill packs dense compute while narrow decode repeatedly streams weights and KV state through HBM.
This is why phase-aware scheduling matters. Decode is carrying a growing memory structure through every loop, so we should stop calling it “the cache” and open it up.
Stage 15 of 31
Lay out the KV cache
Take one accepted green token from the decode loop. Every transformer layer must preserve a key and a value for that position so later tokens can attend to it.
Our illustrative model has 80 layers. Grouped-query attention gives each layer eight KV heads, and each head carries 128 values.
That structure repeats down the layer stack. It is already much larger than the single token tile that entered the stage.
Each illustrative element uses two bytes, and keys need a matching value page. Then the whole paired structure repeats again for the next token.
One token column expands across every layer and head, with a second page for values. The cache grows along the token axis because this entire structure is appended each iteration.
Now every factor is exposed: key and value, layers, heads, head width, element width, and token count. Multiplying them will turn a context limit into actual HBM.
Stage 16 of 31
Calculate KV memory
The picture becomes one product: two for key and value, multiplied by layers, KV heads, head dimension, bytes per element, and total tokens.
For this declared model shape, each additional token adds 327,680 bytes. The model factors stay fixed while generation changes only the token count.
If the reserved sequence grows from 2048 to 8192 tokens, what happens to its KV footprint while every other factor stays fixed?
What happens when context grows from 2K to 8K tokens?
At 2048 tokens the request needs 0.625 GiB. At 8192 it needs 2.5 GiB, exactly four times as much because the token factor is linear.
Switch the Context tokens control through both limits. Compare the per-request HBM reservation, then imagine many admitted sequences growing at the same time.
We can finally join two facts that were separate before: weights occupy HBM continuously, and every admitted 8K sequence needs another 2.5 GiB. Precision therefore changes the HBM-only reservation ceiling, not just model placement.
Stage 17 of 31
Find the HBM reservation ceiling
The same 8K KV reservation now competes with the weights for one 80 GiB device. Select a weight cartridge and follow the HBM budget into a different memory-only ceiling; actual running concurrency can be lower.
Stage 18 of 31
Split layers with tensor parallelism
The precision experiment left us two reasons to use several GPUs: the weights may not fit on one, or we may want more HBM left for active sequences.
Tensor parallelism divides a layer matrix into column shards. Four GPUs can multiply their own slices at the same time, each producing only part of the activation.
This solves the local memory problem, but it creates a distributed result. The next operation often needs the complete activation rather than four unrelated partial vectors.
The workers therefore exchange or reduce their partials inside model execution. Megatron-LM documents this composition and the communication it adds.
All four partial vectors converge into one complete green activation. Tensor parallelism has bought smaller per-device shards by placing a collective inside the repeated layer path.
That collective can happen many times during one token. We can reduce its frequency by splitting a different dimension of the model, but then dependency creates idle bubbles.
Stage 19 of 31
Split depth with pipeline parallelism
Pipeline parallelism leaves each layer whole and assigns consecutive layer ranges to different GPUs. An activation crosses a device boundary only when it reaches the next depth segment.
The first microbatch starts on GPU zero while every later stage waits. Those empty cells are not poor scheduling. They are a bubble created by model dependency.
A second microbatch can enter stage zero as the first moves deeper. More microbatches begin to fill the diagonal with independent work.
Once the pipeline is full, different GPUs process different depth segments at the same time. The fill and drain edges remain unavoidable.
Three microbatches occupy most of the four-device grid, leaving empty cells mainly at the leading and trailing corners. Overlap has reduced the bubble without removing it.
Pipeline parallelism trades activation transfers and bubbles for whole-layer ownership. If we instead copy the complete execution group, independent requests can avoid model collectives altogether.
Stage 20 of 31
Replicate with data parallelism
A data-parallel replica is not another shard of one model instance. It is another complete execution group with its own weights, scheduler, and KV memory pool.
That completeness changes routing. Independent requests can enter different replicas without a cross-replica collective because each group can produce an entire response.
Replicas therefore scale request throughput and isolate failures. The cost is duplicated weight memory, plus another copy to warm and keep compatible during rollouts.
Each replica also owns its cache. Sending a follow-up request elsewhere may throw away prefix or session locality even when the new target is less loaded.
Three request lanes fan into three complete stacks and return separate streams. We have multiplied independent capacity by copying the whole model group.
Inside each copied group, tensor collectives may still run many times per token. Their logical arrangement is not enough. Physical distance now becomes part of latency.
Stage 21 of 31
Place collectives on fast links
Take the four-GPU tensor group we just built. Its workers exchange partials repeatedly, so the path between them is part of every token’s execution.
Placed behind one NVSwitch fabric, the group forms a short local loop. Frequent collectives stay inside the node instead of consuming cross-node bandwidth.
The same logical group can also be scattered across two nodes. Its math remains correct, which makes this placement mistake easy to miss in an architecture diagram.
Now each all-reduce must cross the slower fabric and compete with other transfers. A mathematically valid model layout has become a physically expensive one.
The two equal collective loops reveal unequal distance. One remains inside a fast-link domain, while the other stretches across the red node boundary.
We finally have a valid warm replica with a known HBM budget. The scheduler’s next obligation is to keep that budget valid as every accepted sequence grows.
Stage 22 of 31
Admit by future memory
The placed FP8 replica has a finite green region after weights and runtime buffers. Every active sequence will consume part of that region with KV pages.
If admission counts only current prompt pages, several requests appear to fit. But each one is still allowed to grow toward its declared 8192-token limit.
A fifth request enters before the earlier sequences finish growing. Which admission rule prevents HBM from failing later during decode?
Which rule prevents late HBM exhaustion?
Future reserve expands each request to its allowed footprint before accepting it. The optimistic plan crosses 80 GiB later, while the reserved plan refuses work before memory is promised twice.
Switch the Admission mode through both policies. Compare the HBM verdict, then keep Future reserve selected to solve the capacity challenge.
Admission is where a token limit becomes a memory promise. Once a request is safely inside, the scheduler still has to decide which active sequences share each decode iteration.
Stage 23 of 31
Schedule every iteration
Admission gives us a safe queue, but its requests will finish at different times. A static batch keeps the original membership until the longest sequence ends.
When a short sequence finishes early, its slot becomes an idle hole even though another admitted request is waiting. Memory safety has not guaranteed useful GPU occupancy.
Request B finishes while A still decodes and C waits. What should occupy B’s slot at the next token boundary?
What should occupy the finished slot?
Continuous batching rebuilds membership after each iteration. C and D enter the freed slots immediately, while the static grid preserves empty cells until its original batch ends.
Switch the Batch policy through both choices. Compare the idle holes with the later requests that enter at iteration boundaries.
Iteration-level scheduling removes holes, but it does not make prefill and decode equally shaped work. One long prompt can still occupy the GPU while every active decoder misses its next-token rhythm.
Stage 24 of 31
Chunk long prefills
A newly admitted 4096-token prompt arrives while several sequences are already decoding. Running its prefill as one slab creates a long interval with no green decode tick.
The prompt still has to be processed, so we cannot solve this by deleting work. We can only decide how much of that work is allowed to run before decode gets another turn.
Chunked prefill cuts the long prompt into bounded segments. Each segment retains dense prompt computation without claiming the entire scheduling horizon.
Decode iterations now fit between amber chunks, keeping output-token gaps near their deadline. Sarathi-Serve studies this interference tradeoff for LLM serving.
The single amber slab becomes three pieces with green decode ticks between them. Total prompt work is unchanged, but the longest interval that blocks existing streams is much shorter.
Chunking makes compute time shareable. Memory still has another fragmentation problem because sequences grow and finish at different lengths, leaving holes that rarely form one large contiguous region.
Stage 25 of 31
Page the KV cache
If each sequence reserved one maximum-sized contiguous KV region, short answers would strand large empty tails. Freed regions would also appear in inconvenient sizes and positions.
PagedAttention changes the unit of ownership. Each request receives a logical block table, and each table entry points to a fixed-size physical KV slot.
Those physical slots do not need to be adjacent. Attention follows the table, so request A can own slots two, seven, and one without moving existing KV contents.
The allocator can therefore grow a sequence one block at a time and return blocks individually when it finishes. Contiguity is no longer part of the request contract.
Request B releases two scattered slots, and request C immediately points its table at them. Useful capacity returns without waiting for one large physical hole.
The block table solves physical fragmentation and gives us a reusable unit. If a later request begins with exactly the same tokens, some of those blocks may already contain the right KV state.
Stage 26 of 31
Reuse exact prefixes
Suppose many requests begin with the same long system prompt or document. Their leading token blocks are identical before each user adds a different suffix.
The cache hashes complete token blocks together with the model context that produced them. A hit is valid only when those leading tokens and relevant settings match exactly.
A later request shares the first 3072 tokens but adds a new suffix. Which part of its prompt can skip prefill?
Which region can skip prefill?
The exact leading blocks stay green and reuse their existing KV pages. Only the unmatched suffix enters amber prefill, so three quarters of this illustrative prompt avoid repeated compute.
Switch the Prefix result through both outcomes. Compare the amount of prompt work while the unmatched suffix remains real computation in either case.
A cache hit can shorten prefill, but misses and unique prompts still produce dense prompt waves beside narrow decode loops. We may eventually decide those phases should not share one GPU pool.
Stage 27 of 31
Disaggregate prefill and decode
In one shared pool, every dense prefill competes with active token loops. Chunking manages that conflict, but both TTFT and token-gap objectives still depend on one resource plan.
DistServe takes the stronger step of assigning prefill and decode to separate pools. Each pool can use parallelism and batch shapes suited to its phase.
The split introduces a new object at the boundary: the completed prompt KV state. Decode cannot begin until that state reaches its chosen group.
Placement must make the transfer cheaper than the interference we removed. Otherwise separate pools produce cleaner diagrams and slower first tokens.
Amber prompt waves and green token loops now wait in different queues, joined by a blue KV handoff. Each phase can protect its own latency objective without pretending the boundary is free.
Disaggregation is worthwhile only when saved interference exceeds transfer and operating cost. It also creates two capacity pools whose new replicas still need time to load and warm.
Stage 28 of 31
Scale warm capacity
Demand rises immediately, but a GPU replica does not. It must allocate devices, load weights, prepare collectives, and warm execution shapes before the readiness gate opens.
A scaler that waits for high GPU utilization is already late. Queue age and admitted token load can rise while the new process is still cold.
The capacity target must therefore look forward to when replicas will be ready, not merely count the processes that exist now.
A small warm reserve absorbs the beginning of a burst. Forecasted scaling can then replace that spent headroom while traffic is still being served.
The reactive curve reaches readiness after the queue spike, but the warm reserve bridges the gap. Startup delay has become part of the capacity equation rather than an afterthought.
Warm capacity protects overload, not every failure. A worker can still disappear after two deltas have reached the client, which brings us back to the stream identity we began with.
Stage 29 of 31
Contain streaming failures
Response 42 has already emitted two ordered deltas when its decode worker disappears. Other replicas remain healthy, but those delivered bytes have made this request special.
Routing can remove the failed worker for future traffic. It cannot erase what the client has already read or guarantee that another sampling run will choose identical tokens.
A replacement replica might repeat the two deltas, diverge after them, or use a different random path. Invisible replay would make the response identity lie.
The platform must preserve the partial history the client observed. What should the broken stream do instead of replaying invisibly?
What should happen after a partial stream fails?
Response 42 ends with an explicit terminal error after its two deltas. A retry starts as response 43, with a new creation event and no false claim that the old stream continued.
The API invariant survived a worker failure because client-visible state was treated as real state. To improve the fleet, we now need one trace that connects that outcome to its internal cause.
Stage 30 of 31
Close the operations loop
Response 42 already has the identity we need. OpenTelemetry describes correlated traces, metrics, and logs, so that identity can follow the request across gates, queues, workers, and delivery.
Its spans separate gate time, queue time, prefill, decode gaps, and stream delivery. A slow answer is no longer one undifferentiated latency number.
GPU signals add the missing causes: HBM pressure, active batch occupancy, collective delay, and whether supposedly available replicas were actually warm.
Cost needs the same discipline. GPU hours divided by successful useful tokens exposes idle reserve, failed work, and latency policies purchased with excessive overprovisioning.
One request identity fans into aligned latency, memory, occupancy, error, and cost instruments. A breached SLO can now point back to admission, scheduling, placement, or scaling.
The platform is no longer a fixed stack of components. It is a feedback system that turns one client promise into resource decisions, observes the consequences, and corrects them.
Stage 31 of 31
The serving platform map
We began with what reaches the client: ordered deltas under one response identity, followed by exactly one terminal ending.
That stream exposed two experiences worth protecting separately. TTFT covers silence before text begins, while inter-token delay covers the rhythm after it begins.
Those budgets had to survive the real workload rather than its average. Long prompts, long outputs, model mixture, and synchronized bursts create different kinds of pressure.
Because traffic and fleet policy move at different cadences, a stable control snapshot kept slow placement decisions out of the per-token data path.
The data path then rejected unauthenticated, unfair, or unsafe work before any of it could consume the GPU queue.
For accepted work, routing checked model compatibility first. Only then did warmth, load, prefix reuse, and session locality distinguish valid replicas.
Opening the chosen worker revealed that “the GPU” is not one capacity number. Compute, HBM space, HBM bandwidth, and interconnects can limit different phases.
Weights occupied the first large share of HBM. Fewer bytes per parameter changed both the number of devices required and the room left beside the model.
Merely loading those weights was not enough. The readiness gate waited for kernels, buffers, collectives, and representative prompt and decode shapes to become predictable.
Our chat messages then became the object the runtime truly schedules: one ordered 4096-token prompt with an explicit 512-token output limit and request metadata.
Prefill could process that known prompt as a wide matrix wave, while causal masking controlled information flow and the phase produced the initial KV state.
Decode had to accept one winner, append its KV state, and feed it back before choosing the next token. That dependency created the repeated narrow loop.
The same model therefore turned one GPU into two machines: compute-dense during prefill and often memory-bandwidth-limited during narrow decode.
Inside each decode loop, every accepted token added a key and value across all layers, KV heads, and head dimensions.
Multiplying those factors made sequence length a concrete HBM cost. Our 8K reservation required 2.5 GiB before considering any neighboring request.
Joining that cost with the weight footprint revealed a memory-only ceiling: FP16 overflowed one device, while FP8 and INT4 left room for different numbers of complete 8K KV reservations.
When one device was insufficient, tensor parallelism split layer matrices across GPUs but introduced collectives that rebuilt complete activations.
Pipeline parallelism split model depth instead. It reduced within-layer sharding but created dependency bubbles during fill and drain.
Data parallelism copied complete execution groups, allowing independent requests to run without cross-replica collectives while duplicating weights and caches.
Physical placement then made logical parallelism fast or slow. Frequent tensor collectives belonged inside the shortest high-bandwidth link domain available.
With a valid replica ready, admission reserved each request at its allowed future length. Counting only current pages would have promised the same HBM twice.
Continuous batching reused slots at token boundaries, replacing finished sequences instead of preserving idle holes until the longest original request ended.
Chunked prefill then bounded phase interference. The prompt work remained, but active streams received decode turns between shorter compute segments.
Paged KV allocation solved a different problem by mapping logical request blocks onto scattered physical slots, so freed memory became reusable immediately.
Those blocks also enabled exact prefix reuse. Matching leading tokens could keep prior KV state, while every unmatched suffix still required genuine prefill work.
When shared-pool interference remained too costly, separate prefill and decode pools traded that interference for an explicit KV transfer and more placement complexity.
Scaling those pools meant forecasting future ready capacity. A warm reserve bridged demand while new replicas loaded and prepared their execution paths.
A failed worker finally returned us to the opening contract. The partial stream ended explicitly, and its retry began under a new response identity.
One correlated trace connected that client outcome to queue time, GPU pressure, batch occupancy, failures, warm capacity, and useful-token cost.
The complete design now reads as one chain of promises. The stream defines what users can trust, GPU physics limits what is possible, scheduling allocates those limits, and telemetry teaches the fleet what to change.