How DeepSeek-V4 architecture and training work together
Explore how mHC, hybrid attention, sparse experts, staged pretraining, stability tools, and distillation shape DeepSeek-V4.
The whole story in 6 lines
DeepSeek-V4 couples efficient data paths with a staged, stability-aware training system built for long-horizon work.
- mHC expands the residual stream to four lanes while constrained mixing keeps signal propagation non-expansive.
- CSA selects from four-to-one compressed history while HCA attends densely over a much heavier 128-to-one compression.
- Each Pro token uses six of 384 routed experts plus one shared expert, so only 49B of 1.6T parameters activate.
- Muon handles most weights while guarded routing and SwiGLU clamping address recurring MoE loss spikes.
- Training grows from 4K to 1M tokens and introduces sparse attention at 64K after an indexer warmup.
- Domain specialists learn with SFT and GRPO before full-vocabulary on-policy distillation consolidates them.
The million-token pressure test
An agent keeps a million tokens of code, tool results, and decisions in one working history. Ordinary dense attention makes every new token compare against that expanding past.
Four ideas will anchor the journey. Residual streams carry state through layers, attention reads context, experts provide sparse compute, and distillation combines trained behaviors.
The architecture rail changes how signals, memory, and expert compute move. The training rail teaches those mechanisms gradually and protects the run from instability.
This is a co-design story grounded in DeepSeek’s preview report and official code. Now let us start with the signal highway connecting every Transformer block.
mHC keeps four residual lanes bounded
We begin with the path that carries information between layers. DeepSeek-V4 expands one residual stream into four lanes, giving each block more ways to combine old state with new computation.
Each block reads a learned mixture of those lanes and writes its result back through another learned mixture. That extra freedom is useful, but unconstrained mixing can amplify signals across a deep stack.
DeepSeek therefore projects the residual mixing matrix onto a special surface. Which property keeps repeated mixing from expanding the signal without bound?
The four lanes now cross a doubly stochastic mixer. Signal can redistribute among lanes, but the residual transformation remains non-expansive through repeated layers.
mHC adds a capacity axis while constraining the path that carries it. The published design applies Sinkhorn projection for twenty iterations. That does not prove every stability gain alone, but it gives the deeper model a bounded transport layer. Next, we will compress its memory path.
Two compressed views of one history
The residual lanes can carry state, but attention still has to read a million-token history. DeepSeek-V4 reuses one token tape through two different compressed views instead of applying dense attention everywhere.
Compressed Sparse Attention pools every four tokens into one entry. A lightning indexer then selects 1,024 compressed entries for each Pro query, while a 128-token local window preserves nearby detail.
The second path compresses history by 128 times and removes the sparse selector. After such heavy compression, what attention pattern remains practical?
Both paths now feed the same query. CSA keeps finer compressed blocks and chooses a relevant subset, while HCA keeps a much shorter global summary and attends across all of it.
Interleaving CSA and HCA trades detail, coverage, memory, and compute across layers. Their order across layers shapes the final cost profile. The report attributes its million-token efficiency to this hybrid plus low-precision storage. Next, we will examine sparse compute inside each block.
A huge expert bank with a narrow active path
Compressed attention controls the memory bill, but each Transformer block still needs feed-forward compute. DeepSeek-V4-Pro places 384 routed experts and one shared expert inside every MoE layer.
The router scores the token against that bank. Only six routed experts receive the token, while the shared expert supplies a common path that every token can use.
The full checkpoint contains 1.6 trillion parameters. How much of that model does one Pro token activate during its forward pass?
Six bright routes cross the larger expert field and rejoin the shared stream. That sparse path activates about three percent of the total Pro parameters for one token.
Sparse experts separate model capacity from per-token compute, but routing also creates uneven loads and outliers. The same sparse choice changes how imbalanced training failures can appear over time. Those training risks are part of the architecture. Next, we will follow one loss spike and its recovery.
Stop a loss spike from returning
Sparse compute must remain trainable for tens of trillions of tokens. DeepSeek-V4 applies Muon to most matrix weights, while AdamW remains responsible for embeddings, prediction heads, RMSNorm weights, and mHC gates.
The team still observed loss spikes tied to MoE outliers, with routing apparently making them worse. Restoring an earlier checkpoint resumed training, but the same conditions could trigger another spike.
A rollback has returned the loss curve to its earlier value. Which response changes the path that produced the spike?
The guarded branch uses cached historical routing indices after a spike and clamps extreme SwiGLU values. The loss trace continues instead of looping through the same broken region.
Switch the Recovery control between both choices. Notice whether the trace returns to another spike or passes through a bounded recovery.
The report says these techniques stabilized training without hurting performance, but it leaves their full theory open. That boundary matters. Next, we will see how the context curriculum avoided another abrupt transition.
Grow context before turning sparsity on
★ If you remember one thing · DeepSeek-V4 reached one million tokens by growing context in stages and introducing sparse attention only after shorter dense training.
Now that the run can recover from instability, the attention system still needs a learnable path to extreme length. Pro training begins with 4K sequences and dense attention.
The sequence grows to 16K while the model keeps the simpler dense regime. This phase lets the backbone learn longer dependencies before the sparse indexer must choose among many compressed blocks.
At 64K, the report introduces sparse attention after a short lightning-indexer warmup. What changes when the ribbon crosses that gate?
The trained sparse selector stays attached as context expands to one million tokens. The model reaches the longest ribbon by carrying forward mechanisms learned at shorter lengths.
DeepSeek reports the four sequence lengths and the 64K sparsity transition. It does not publish the exact duration of Pro’s initial dense stage. Token counts and compute spent in the remaining stages are also undisclosed. Next, we will turn the pretrained base into one capable assistant.
Many specialists teach one student
The pretrained base now knows broad patterns, but useful behavior needs targeted practice. Separate math, code, agent, and instruction specialists each begin with domain SFT and continue with GRPO reinforcement learning.
Those specialists do not simply merge their weights. A single student generates its own trajectories, then the relevant teachers evaluate complete next-token distributions along that student-created path.
A cheaper shortcut would keep only one sampled token from each teacher. Why does the published pipeline preserve the full vocabulary distribution instead?
More than ten teachers now converge on one student through full-vocabulary on-policy distillation. The final model consolidates specialized behavior without physically carrying every teacher at inference time.
Post-training also adapts MoE expert weights and the CSA indexer path to FP4 deployment through quantization-aware training. The exact teacher blend remains an unpublished training choice. Architecture, training, and serving remain connected. Now let us step back and see the whole picture together.
The co-designed system
We started with mHC, where four residual lanes add capacity while constrained mixing keeps the transport non-expansive across deep stacks.
Then CSA and HCA gave one long history two efficient views. They combined selective detail with a compressed global summary.
Sparse routing separated total capacity from active compute, sending each Pro token to six routed experts and one shared expert.
Muon handled most matrix updates. Anticipatory routing and SwiGLU clamping answered recurring MoE loss spikes.
The context curriculum grew through four lengths and introduced sparse attention at 64K, allowing the indexer to learn before the million-token stage.
Specialist SFT and GRPO produced domain teachers, then full-vocabulary on-policy distillation consolidated their behavior into one student.
Together, these mechanisms form one co-designed system for long-horizon intelligence. The published report confirms the pieces, while several costs, mixture details, and causal ablations remain unknown.
The whole story in 6 lines
DeepSeek-V4 couples efficient data paths with a staged, stability-aware training system built for long-horizon work.
- mHC expands the residual stream to four lanes while constrained mixing keeps signal propagation non-expansive.
- CSA selects from four-to-one compressed history while HCA attends densely over a much heavier 128-to-one compression.
- Each Pro token uses six of 384 routed experts plus one shared expert, so only 49B of 1.6T parameters activate.
- Muon handles most weights while guarded routing and SwiGLU clamping address recurring MoE loss spikes.
- Training grows from 4K to 1M tokens and introduces sparse attention at 64K after an indexer warmup.
- Domain specialists learn with SFT and GRPO before full-vocabulary on-policy distillation consolidates them.