Why PI-conditioned self-distillation can fail on reasoning tasks
Explore how privileged-information bias can decouple self-distillation loss from correctness and suppress useful reasoning paths.
Lesson summary
Dense supervision is only useful when its target distinguishes correctness from imitation of one privileged trajectory.
- The self-teacher sees privileged answer information, while the student matches its token distribution without seeing that help.
- On easy recall, imitation can track success; on difficult reasoning, loss falls while validation usually stays flat or declines.
- A full solution makes the teacher favor one trajectory, while weaker hints and skills move too little probability to teach reliably.
- Correct and incorrect rollouts receive nearly the same loss and KL, so the objective cannot direct effort toward failures.
- Low-information tokens absorb 55.38 percent of the measured loss share, crowding out answer-determining decisions.
- A productive detour in a correct rollout receives about four times the KL of a reference-aligned position.
- The average teacher-student gap reaches zero without its spread shrinking, while entropy rises and accuracy does not.
What makes this self-distillation signal dense?
The student is trained against the PI-conditioned teacher distribution at every token position along its own rollout.
What must be checked before treating a falling self-distillation loss as learning?
Validation accuracy must improve too. On the difficult tasks in the paper, it usually stayed flat or declined.
What does a large gap between the shown solution and another correct solution mean?
The teacher is transferring preference for one trajectory rather than a general signal for correctness.
How did the loss behave for correct and incorrect rollouts?
Mean per-token loss and mean KL overlapped, and the teacher-student gap was actually larger on correct rollouts.
Which token groups absorbed most of the total per-token loss?
Stopwords, uncertainty markers, punctuation, and whitespace together absorbed 55.38 percent.
Why can a correct reasoning detour receive a large update?
The teacher has read one reference path and cannot tell a productive alternate step from a mistake merely by divergence from that path.
Why does an average teacher-student gap near zero not prove convergence?
The token-level spread remained wide and student entropy rose, so positive and negative disagreements were cancelling rather than disappearing.
Stage 1 of 9
Setup
A training run reports exactly what we hope to see: its loss slides downward step after step. Then validation arrives, and the model solves no more difficult problems. In several settings it solves fewer.
The student is the model being updated. Its self-teacher is the same model, or a slowly moving copy, allowed to read extra answer information that the student will not receive at test time.
That extra clue is privileged information, or PI. A divergence measures how far the student’s next-token distribution sits from the PI-conditioned teacher’s distribution.
We will build the dense objective, reproduce its split training signature, manipulate the PI target, and then follow the misplaced pressure through rollouts, token types, exploration, and the final student.
The mismatch is now the question, not a footnote. A lower loss only proves that the student matched its chosen target more closely. Let us find out what that target actually rewards.
Stage 2 of 9
The dense self-distillation objective
The student begins with only the problem and generates its own reasoning rollout. This matters: training evaluates the model along the sequence it actually produced, including detours and mistakes.
A second forward pass uses the same prefix but also includes a reference solution, hint, or feedback. That privileged context makes the self-teacher a stronger predictor of one plausible continuation.
The teacher distribution is compared with the student distribution at every position. Does this objective ever check whether the completed rollout is correct?
Does the lone self-distillation objective check whether the completed rollout is correct?
No correctness signal enters this loss. Stop-gradient freezes the teacher target for the update, so gradients flow only through the student as it is pushed toward the privileged distribution.
The completed picture exposes the bargain. One reference example becomes dense per-token supervision without an external teacher, but correctness sits outside the route that supplies the gradient.
Density solves the scarcity of feedback, not the meaning of the target. If imitation and correctness line up, that distinction can hide. The next comparison shows when they separate.
Stage 3 of 9
When falling loss stops meaning learning
On short SciKnowEval questions, the authors reproduce the earlier result: loss falls and validation rises. One reference answer nearly covers the small space of successful responses.
They then keep the implementation and apply the lone objective to general QA, mathematics, coding, and multi-turn tool use. These tasks allow longer reasoning and more than one useful route.
Optimization remains smooth in the harder regime, so the code is clearly reducing its objective. What happens to validation accuracy when imitation no longer stands in for correctness?
What happens to validation accuracy on the harder tasks?
The two gauges now disagree. Loss continues downward, while validation stays flat or moves backward; entropy rises and responses shorten. Together those changes form the paper’s failure signature.
Switch the Task regime control through both options. The loss curve keeps improving, but the validation verdict changes from aligned progress to a decoupled proxy.
A falling self-distillation loss is therefore not enough evidence of learning. The mechanism begins one step earlier, inside the privileged target itself.
Stage 4 of 9
PI bias: one solution is not correctness
Select the Full solution, Hint, and Skill slips. Follow the teacher’s two beams into equally correct answers, then read the measured average change beneath the verdict. Which PI actually encodes correctness?
A full reference solution pulls the teacher toward that one trajectory, not toward correct solutions in general.
Stage 5 of 9
The loss cannot find the failures
The PI Bias Score showed that the teacher prefers one route. We can now ask whether its per-token loss still happens to push harder on rollouts that end incorrectly.
The authors split student rollouts by verifier outcome. Mean per-token loss settles near 3.5 times ten to the minus four for both groups, and mean KL overlaps near 0.03.
A useful objective should reserve more pressure for mistakes. When the teacher-student log-probability gap finally separates, which group receives more?
Which group retains the larger teacher-student gap?
The direction is reversed. Correct rollouts retain the larger gap, so the strongest learning pressure lands where the student already succeeds instead of concentrating on its failures.
An objective blind to outcome can fall without task accuracy following it. The next question is where all that dense token-level pressure goes.
Stage 6 of 9
The loss budget lands on the wrong tokens
Dense supervision sounds valuable because every position can teach. But every position includes words and marks that make reasoning readable without deciding whether the answer is right.
The paper labels nine token types. Uncertainty markers such as “wait” and “maybe” carry the highest average per-token loss, followed by stopwords, while numbers and special tokens rank near the bottom.
Now add the total pressure absorbed by stopwords, uncertainty markers, punctuation, and whitespace. Do these low-information groups remain a small side cost?
How much of the total per-token loss do the low-information groups absorb?
They take 55.38 percent of the budget. The objective spends most of its capacity matching hesitation, connective words, punctuation, and spacing rather than the content decisions that fix the answer.
The teacher’s surface preference is now consuming real optimization effort. Worse, the largest divergence can occur precisely when a correct solver explores away from the shown route.
Stage 7 of 9
A correct detour looks like an error
A difficult solution often tests a partial idea, backs up, and then reaches the answer. That search is not noise added after reasoning; it can be how the answer is found.
The paper ranks positions by how strongly the student prefix supports the reference solution. Inside correct rollouts, the top quartile is called on-path and the bottom quartile off-path.
Both branches end at the same verified answer. Which one does the PI-conditioned teacher punish more strongly?
Which correct branch receives the larger divergence?
The productive detour receives roughly four times the KL: about 0.31 versus 0.08. The teacher can recognize distance from its reference, but not whether that distance is useful exploration.
Repeated updates make hesitation expensive, so responses shorten and the model commits earlier. The final link is the change left in the whole next-token distribution.
Stage 8 of 9
The student flattens instead of sharpening
The exploration tax predicts a more decisive student, but the distribution tells a subtler story. The signed teacher-student gap begins near 3.1 and moves toward zero during training.
Around step fifteen the mean undershoots to about minus 0.65, then settles near zero from roughly step thirty. Read alone, that average could look like successful matching.
If the student truly converged to its teacher, what should happen to disagreement at individual token positions?
What should genuine convergence do to the token-level spread?
The spread does not contract; it remains near plus or minus one while the mean cancels to zero. Student entropy rises, the teacher finds student rollouts harder to predict, and the policy ends flatter rather than sharper.
The full signature now fits one mechanism: earlier commitment, shorter reasoning, and no validation gain. The paper does not show that every self-distillation method fails; it shows that this PI-conditioned per-token target alone is misaligned with correctness.
Stage 9 of 9
Recap
We began with a dense self-distillation objective. The student sees the problem, the self-teacher also sees privileged answer information, and every token position contributes divergence.
Then the easy and difficult regimes separated. A falling loss accompanied better validation on short recall tasks, but not across the harder reasoning domains.
The PI Bias Score identified the first break. A full solution pulled probability toward its own trajectory, while hints and skills weakened the whole signal rather than revealing correctness.
Outcome labels did not rescue the objective. Correct and incorrect rollouts received overlapping loss and KL, with greater remaining pressure on the already-correct group.
At token level, low-information words and marks absorbed 55.38 percent of the measured loss share, leaving less pressure on answer-determining content.
Inside correct reasoning, distance from the shown path became expensive. Productive off-path positions received roughly four times the KL of on-path positions.
Finally, the mean teacher-student gap reached zero without the spread shrinking. Entropy rose, responses shortened, and validation did not improve.
These measured values came from the paper’s Qwen3 runs, not a fabricated toy trace. The practical lesson is precise: monitor validation, screen PI bias, and do not mistake a better proxy for a better reasoner.