How DeepSeek V4 million-token context architecture works
Explore DeepSeek V4 Pro and Flash, CSA, HCA, mHC, sparse MoE routing, Muon pretraining, and on-policy distillation from primary sources.
The whole story in 6 lines
DeepSeek V4 makes million-token work practical by co-designing compressed attention, sparse capacity, stable depth, and training.
- V4 Pro and Flash are documented preview models, while real deployment behavior beyond published tests remains uncertain.
- At one million tokens, DeepSeek reports V4 Pro uses 27% of V3.2 inference FLOPs and 10% of its KV cache.
- CSA compresses overlapping groups fourfold, then selects a small relevant subset plus recent uncompressed tokens.
- HCA compresses 128 tokens into one entry and reads all compressed entries, complementing selective CSA layers.
- mHC stabilizes four residual streams while DeepSeekMoE activates six routed experts and one shared expert per token.
- V4 grows context in stages, introduces sparse attention at 64K, then merges specialist skills with on-policy distillation.
Setup
Imagine giving a model a million-token project archive. Ordinary attention compares each new token with an enormous history, so both computation and memory become the problem.
Context means the tokens currently available to the model. The KV cache stores attention memories from earlier tokens so generation does not recompute everything.
Sparse attention reads only selected history, while an expert is one specialized feed-forward network inside a larger mixture. These ideas reduce different kinds of work.
Our route now separates six questions: what is published, where cost grows, how CSA and HCA compress memory, how blocks compute, and how training joins everything.
You now have the four terms needed for the tour. Let us begin with the primary evidence and identify exactly which V4 models DeepSeek released.
Evidence and Family
The setup gave us a million-token problem. Before explaining solutions, we need an evidence ledger because model announcements often blend architecture, measured claims, and speculation. That separation keeps inspectable release artifacts distinct from measurements reported by the model maker and from claims the available sources cannot establish.
DeepSeek released V4 as a preview on April 24, 2026. The official collection provides Pro and Flash model cards, open weights, configuration files, and reference inference code.
Flash has 284 billion total parameters with 13 billion active per token. Pro has 1.6 trillion total parameters with 49 billion active per token.
The evidence now divides cleanly. Architecture settings come from inspectable primary artifacts, while efficiency and benchmark numbers remain DeepSeek-reported measurements from its own evaluation setup.
Unknowns still include performance on workloads that were not published and independent production reliability at full context. Next, we will examine why that long context is costly.
The Million-Token Cost
Both architectures receive the same million-token history. The long row stays fixed because the experiment changes storage strategy rather than shrinking the user input. The cache footprint is the comparison, not the context limit.
The V3.2 baseline keeps a much larger attention memory. V4 instead mixes recent uncompressed entries with compressed memories that occupy far fewer stored positions.
The context length does not change, but the stored memory can. Which architecture should leave the smaller KV-cache footprint?
DeepSeek reports that V4 Pro needs 27 percent of V3.2 single-token inference FLOPs and 10 percent of its KV cache at one million tokens.
Switch the Architecture control through both choices. Compare the downstream memory column and cost meter while the million-token input remains exactly the same.
Compression changes the cost without changing the promised context length. Next, we will open CSA and see how it compresses history before selecting relevant entries.
Compressed Sparse Attention
The cost comparison showed that V4 stores less long-range memory. CSA begins by turning overlapping groups of four token memories into weighted compressed entries. The compression is learned rather than fixed, so each stored summary can preserve useful boundary information while still reducing the number of long-range positions.
Overlap lets neighboring compressed entries share boundary information. A learned weighting operation decides how strongly each original token contributes instead of using a plain average.
Compression alone still leaves many old entries across a million-token history. What should the lightning indexer pass into core attention?
The indexer keeps the strongest compressed entries, up to 1,024 for Pro and 512 for Flash. A separate 128-token uncompressed window preserves fine local detail.
CSA reduces long-range work twice, first through fourfold compression and then through sparse selection. Next, HCA will trade more detail for a much smaller global memory.
HCA and the Hybrid
★ If you remember one thing · DeepSeek V4 alternates selective 4-to-1 memory with dense 128-to-1 memory, while both keep a local 128-token branch.
CSA preserves more long-range detail and chooses a sparse subset. HCA takes the opposite bargain by compressing each non-overlapping group of 128 tokens into one entry. This creates complementary memory scales: a detailed selective path and a tiny dense path that can be read completely.
Because HCA creates so few entries, it can attend densely across all of them. It also keeps the same 128-token local branch for precise recent relationships.
One lane keeps detail and selects sparsely, while the other compresses heavily and reads densely. How should V4 combine them across layers?
The hybrid now appears side by side. CSA keeps a medium fourfold memory and selects top entries, while HCA keeps a tiny 128-fold memory and reads every entry.
DeepSeek interleaves these attention types after the opening layers, with different starts for Pro and Flash. Next, we will inspect what happens around attention inside each block.
Residuals and Experts
Hybrid attention explains where a token reads. A V4 block must also carry information safely through depth, so mHC expands one residual stream into four parallel streams. These residual streams change signal transport, while expert routing changes how much conditional capacity each token can use.
Input-dependent mappings mix those streams before and after each layer. Sinkhorn normalization constrains the residual mapping to a doubly stochastic matrix, which bounds signal expansion.
The token then enters DeepSeekMoE. Pro offers 384 routed experts and Flash offers 256, while both select six routed experts plus one shared expert per token.
Four stable residual paths braid into one token, which then fans through only a few bright expert cells before their outputs merge. Depth and capacity scale separately.
mHC protects signal flow while sparse experts add capacity without activating the full model. Next, we will see how training introduces long context and then consolidates specialist skills.
Training and Distillation
The block architecture is only useful after stable training. DeepSeek reports 32 trillion pretraining tokens for Flash and 33 trillion for Pro, using Muon for most parameters.
Training starts with 4K-token sequences, then grows through 16K and 64K before reaching one million. This curriculum avoids beginning with the hardest memory regime.
Flash begins with dense attention for its first trillion training tokens. Sparse attention enters at 64K after a short indexer warmup, then remains active.
After pretraining, separate specialists learn reasoning, coding, agent behavior, and instruction following through supervised tuning and GRPO reinforcement learning.
On-policy distillation brings more than ten specialist teachers toward one student using full-vocabulary reverse-KL targets. The final model receives their skills without serving them separately.
The published path joins long-context pretraining, specialist improvement, quantization-aware training, and distillation. Now let us step back and see the whole evidence-backed system together.
Recap
We started by separating primary artifacts from DeepSeek-reported measurements and open unknowns. That evidence boundary keeps the architecture useful without overstating independent verification.
Then we held one million tokens constant and changed the storage design. DeepSeek reports that V4 Pro sharply reduces both single-token compute and KV-cache memory.
CSA compressed overlapping groups and selected relevant history, while its local window preserved recent detail. Two reductions made selective long-range recall affordable.
HCA compressed much more aggressively and read its tiny memory densely. Interleaving HCA with CSA gave different layers complementary views of the same context.
Inside each block, constrained residual streams stabilized depth and sparse experts expanded capacity. The token used only six routed experts plus one shared expert.
Training grew sequence length step by step, introduced sparse attention at 64K, and later consolidated specialist models through full-vocabulary on-policy distillation.
The full picture is one co-design across memory, computation, residual transport, routing, and training. These numbers come from official primary artifacts, while broader behavior still needs independent testing.
The whole story in 6 lines
DeepSeek V4 makes million-token work practical by co-designing compressed attention, sparse capacity, stable depth, and training.
- V4 Pro and Flash are documented preview models, while real deployment behavior beyond published tests remains uncertain.
- At one million tokens, DeepSeek reports V4 Pro uses 27% of V3.2 inference FLOPs and 10% of its KV cache.
- CSA compresses overlapping groups fourfold, then selects a small relevant subset plus recent uncompressed tokens.
- HCA compresses 128 tokens into one entry and reads all compressed entries, complementing selective CSA layers.
- mHC stabilizes four residual streams while DeepSeekMoE activates six routed experts and one shared expert per token.
- V4 grows context in stages, introduces sparse attention at 64K, then merges specialist skills with on-policy distillation.