Can the same layers think again?
Solve a small puzzle, change the return path, and discover what another pass can actually buy.
Stage 1 of 9
Why run it again?
If a model needs more time to answer a difficult question, could we give it another run through the same layers? There is an obvious objection: doing the same calculation again ought to give us the same result. We can examine that objection with a small machine trying to work out whether a message can travel from A to D.
The route runs from A through B and C to D. We can see it, but our machine begins with only the starting place, A, marked. A filled circle records a place it has established as reachable; an empty circle records one it has yet to establish. Those marks are its state, and the task is to extend what they tell it until it can settle the question about D.
To extend that state, the machine keeps its existing marks and adds places one connection away from them. We call one application of this rule a pass. The important detail is which marks a pass is allowed to read.
Stage 2 of 9
What one pass can know
It takes a snapshot before checking the connections and writes any additions into a separate result. Otherwise, a mark created while checking one arrow could immediately feed another, and the order of the checks would change how much work a pass accomplished.
On the arrow from A to B, the snapshot contains the mark we need: A is already reachable, so B can be added to the result. The other arrows get the same test against the same starting snapshot.
After checking every arrow against the starting snapshot, which places will be marked?
Which places are marked after one pass?
As the checks finish, the result contains A and B. B has acquired a mark, but the snapshot we were reading still had B empty, so it could not help us reach C during this pass. We have learned something that the calculation could use, just too late for the calculation we have already done. It would have to become part of the input to another pass.
That leaves us with a choice about what to send back through the rule. Sending the original state, containing only A, would throw away the discovery about B. Sending the result would let the next pass start with both marks.
Stage 3 of 9
What comes back into the loop?
The Original input and Previous state connections let you try those two choices with exactly the same rule. With Original input, every pass produces A and B because every pass starts with only A. With Previous state, the results are A and B, then A, B and C, then all four places. The mark we added at B becomes usable on the next pass, which adds C in time for the pass after that to reach D. Repetition helps because the calculation receives information that an earlier run did not have.
The rule stays the same. Feeding back its updated state lets the next pass build on earlier work.
Stage 4 of 9
What would a model carry forward?
To carry that idea into a model, we need something that can retain the work of an earlier pass. The model's hidden state, an array of numbers, can play that role. Those numbers carry information, although they do not have the meanings we assigned to our marks. We cannot point to an arbitrary component and read it as “B is reachable.”
The state goes through a group of layers, called a block, which uses learned numbers called weights to transform it. In the recurrent design we are following, the original question, also encoded as numbers, remains available to that block. Its output is an updated state that can return through the same layers, so the next pass can receive different information while using the same weights.
That explains how another pass can produce something different. Whether the difference helps is the harder question, and it takes us to how the block was trained. In the recurrent depth study, training ran the shared block for varying numbers of passes.
Errors in the resulting text predictions guided changes to its weights, giving training a way to learn updates that work when repeated. Such training provides a reason to investigate additional passes; it does not guarantee that every additional pass improves an answer. When the trained model answers a new question, those weights stay fixed while the state changes.
After the allotted passes, output layers read the final state and predict a token, a piece of text. Our illustration produces “Yes.” That is the first time we have needed to turn the intermediate work into text.
Stage 5 of 9
Does thinking have to print words?
If we wanted a written scratchpad instead, the model would generate intermediate tokens that later computation could read. In our route example, those tokens might describe the steps along the path. Writing each addition would lengthen that text sequence, leaving a growing record for subsequent work to use.
The recurrent path carries the work forward without that conversion between passes. Each new version of the hidden state becomes available to the next calculation directly.
This is the sense in which the computation is latent: the intermediate work stays in the internal representation until we ask the output layers for text. “Neuralese” is an informal name for that possibility, not evidence that the numbers contain a secret sentence. The two paths can therefore end with the same short answer after doing their intermediate work in different forms.
We may avoid generating scratchpad tokens by carrying state directly, but each trip through the recurrent block still requires a calculation. To understand what repeating it costs, we have to distinguish keeping its weights in memory from using them again.
Stage 6 of 9
Where the extra work is paid for
Our illustration stores eight layers: two at the input, four in the recurrent block and two at the output. Those four middle layers do not need another copy of their weights each time we use them.
On a single pass through the block, however, every stored layer runs once, so eight stored layers also happen to mean eight layer applications.
Keep the same stored layers, but run the middle four three times. What changes?
What changes with three recurrent passes?
The execution record now contains two applications at the start, twelve through the middle and two at the end. That is sixteen applications of the same eight stored layers. Reusing the weights has let us make the calculation longer without storing a larger set of layers, but we are still paying for those additional uses.
This count does not tell us elapsed time; it makes clear why deciding how many passes to run is a separate choice from deciding how many layers to store.
Stage 7 of 9
When nothing changes
Once another pass costs us work, we need a way to decide when to stop. One possibility is to compare successive states and stop when they no longer change. Our Original input experiment gives us a chance to test that idea: its third and fourth results both contain only A and B, so the comparison reports no change.
The experiment using Previous state eventually reports no change too, with all four places marked in both its third and fourth results. From the comparison alone, both processes appear to have settled.
Does an unchanged result alone tell us that the answer is correct?
Does an unchanged state prove the answer is correct?
Checking D reveals what the comparison missed. Only the process using Previous state reached it; the other settled while a reachable destination was still unmarked. Matching successive results tells us only that the recorded output has stopped changing, not that the task was solved.
For a model, we would therefore need to test a stopping signal against answer quality before trusting it to decide when enough work had been done. We could choose a fixed pass budget instead, but even our tiny example gives us a reason to question a single budget for every task.
Stage 8 of 9
Spend the fewest passes
Change the connections so that A leads directly to both B and C, while C still leads to D. There are still four places and the rule is unchanged, yet the work needed to establish a route may have changed.
Each pass still has to read the marks it started with before its new marks can help a later pass. Follow that restriction through the changed network: the current result shows how far the selected budget gets us.
Adjust Pass budget to the smallest value that reaches D. Compare the snapshots to see whether any pass adds nothing.
Two passes suffice here. The first reaches B and C together, and the second can use C to reach D. Checking the branch to B adds no separate pass because that check can happen alongside the one to C. A third pass leaves the marks unchanged. The route's dependencies, rather than the number of places alone, determined how much repeated work was useful.
Stage 9 of 9
What would convince us?
That gives us a concrete way to think about giving a model more time.
In the puzzle, we could inspect what one pass established and follow that information into the next, so the benefit of repetition was visible.
Inside a learned model, the state does not give us that same readable account. We have to establish the benefit by testing whether additional passes improve its answers on the problems we care about.
Those answers might be just as short with either budget, so their length cannot tell us how much work preceded them.
We need to measure the work as well as the result, including the time spent reusing the same weights.
If a stopping rule chooses the budget for us, it needs the same scrutiny: settling is useful only insofar as it lets us stop without sacrificing the answer we needed.
A claim about “more thinking” becomes useful when it tells us what that extra work achieved and what it cost. This example does not establish GPT-6's architecture.
Research behind the lesson
The reachability puzzle and layer budget are teaching examples. They illustrate mechanisms from these papers; they do not simulate a trained model or establish GPT-6’s architecture.
- Scaling up Test-Time Compute with Latent Reasoning
A shared recurrent block can spend extra inference computation before decoding.
- Reasoning with Latent Thoughts: Looped Transformers
Theoretical and experimental work on repeated computation, including iterative tasks.
- Universal Transformers
Repeated attention-based updates and adaptive computation per position.
- Coconut: Reasoning in a Continuous Latent Space
A related but different mechanism: feed a hidden state back as the next input embedding.
Lesson summary
Repeated computation is useful when each pass receives a useful new state; shared weights save storage, not the cost of doing work.
- Each pass reads the previous state; a newly reached place contributes on the next pass.
- Reusing a rule can make progress because the state fed into it changes.
- Recurrent Transformers reuse learned weights across depth while updating continuous hidden states.
- Latent computation can continue without emitting an intermediate word token.
- More passes add layer applications while the stored weights stay fixed.
- An unchanged state is evidence of stability, not proof of a correct answer.
- A useful compute budget depends on the input and the update rule, not just the size of the diagram.
Why can the first pass reach B but not C in the worked chain?
Only A was reachable at the start of that pass. B can carry the information onward on the next pass.
Why does repeating the same rule sometimes produce new information?
The rule receives the updated state from the previous pass. Repeating it on the original input discards that progress.
Which changes between passes: the hidden state or the shared weights?
The hidden state changes. The learned weights stay fixed while the recurrent block is applied again.
Does another latent pass require another word in the context?
No. A recurrent block can update hidden states without sampling a scratchpad token, then decode an output later.
How many layer applications does our eight-layer model use for three passes?
Sixteen: two input layers, twelve recurrent applications, and two output layers. It still stores eight distinct layers.
Why does a stable prediction not guarantee a correct answer?
A process can repeatedly return the same incomplete or wrong result. A stopping signal must be evaluated for answer quality as well as cost.
Why did the new network need fewer passes despite having four places?
The route from A through C to D had only two connections. The branch through B did not make that route longer.