J-space: Testing Unspoken Ideas
Read an internal concept, open the mathematics of a swap, and follow the experiment that tests whether it changes the answer.
Stage 1 of 13
The missing intermediate
Ask a language model how many legs the animal that spins webs has. It answers 8. The word spider appears in neither the question nor the answer, yet identifying a spider is a useful intermediate step. Did anything like that happen inside the model?
The researchers in this study found a spider-like internal representation and intervened on it. Three terms will help us open their experiment. An activation is a list of numbers inside the model. A lens turns that state into a readable view. An intervention deliberately changes the state to test what it affects. An answer alone cannot tell us which intermediate steps produced it.
We will keep that animal question beside a small numerical example. The numbers let us calculate the measurement and edit ourselves. They are invented, and the experiment’s reported answers will be labeled separately. Our task is to connect something readable inside a model to something the model actually uses.
Stage 2 of 13
Where can an unspoken idea live?
A transformer processes text in tokens, which can be words or pieces of words. At each token position, it carries a vector: a list of numbers. Layers update that vector as they combine information from the context. This running state is called the residual stream.
Follow one position through our toy layers. The starting vector is [0.20, 0.10, 0.40]. Adding the first layer’s update, [0.60, 0.10, 0], gives [0.80, 0.20, 0.40]. The next update produces [1.12, 0.16, 0.40]. The same three slots persist while their values change.
Those slots are not three named ideas. A representation can point along a combination of slots, and different representations can overlap. We need a way to relate an intermediate vector to words without pretending that one neuron simply means spider.
At the final layer, the model already has a learned vocabulary readout. After normalization, that readout turns the final vector into token scores. An earlier vector uses a different geometry. Applying the final readout directly can therefore miss what the earlier state is positioned to express. We need a map between the two.
Stage 3 of 13
What does a Jacobian measure?
To build that map, start with a smaller question: if we nudge one number in an intermediate activation, how does the final activation change? Hold the context fixed and compare the original run with the nudged run. Their difference exposes sensitivity to that particular input direction.
Our two-dimensional toy map makes the calculation visible. Increase the first input component by 0.10. In this example the final vector changes by [0.02, 0.08]. Dividing by the nudge gives [0.20, 0.80]: the output change per unit input. Those two numbers fill the first column of the map.
Nudging the second component instead gives a different output change, [−0.08, 0.01]. Divide by 0.10 again and the second column is [−0.80, 0.10]. The completed matrix records how each output component responds to each input component. This local sensitivity map is a Jacobian; the researchers compute derivatives, while our finite nudges illustrate what they measure.
Use the component selector to compare the two probes. The input arrow changes direction, and the corresponding output change supplies one column. We have exposed a local relationship, not replaced the remaining network with a universally exact linear function.
Stage 4 of 13
Why average the sensitivity maps?
That Jacobian depends on the context. A spider representation might contribute to a number in a legs question and to a word in an animal-name question. A map fitted to only one continuation could confuse that particular use with the representation’s broader capacity to be expressed.
The researchers average Jacobians over many contexts and over source and same-or-later destination token positions. They build a separate average for each layer. The aim is a general readout of what a state is disposed to express, rather than a prediction tailored to the current continuation.
Here three invented contexts each supply a matrix. Average corresponding cells, not whole rows shuffled together. The upper-left entries, 0.20, −0.20, and 0, average to 0. The other cells produce the shared map shown on the right. The real study averages over 1,000 contexts in a much larger activation space.
Apply our average map to the first two numbers of the animal example, [1.12, 0.16]. Each output is a row’s weighted sum: 0 × 1.12 − 1 × 0.16 gives −0.16, and 1 × 1.12 + 0 × 0.16 gives 1.12. The third component is omitted from this two-dimensional demonstration. We now have a mapped vector in the final readout’s coordinate system.
Stage 5 of 13
Is a lens word the next output?
Normalize that mapped vector, then compare it with the vocabulary directions. In our toy readout, the spider direction receives a larger score than the ant direction. The ranked words give us a vocabulary for inspecting an activation that was originally just numbers. This average-map-plus-vocabulary readout is the Jacobian lens, or J-lens.
Keep the lens on a side branch. The actual model still runs its remaining layers and produces the answer 8. A high spider lens score says that this representation is poised to express spider across contexts. It does not say spider is the next token the model will print in this question.
We can combine the two readout operations into one. Multiplying each vocabulary row by the average map gives a token-associated direction back in the original layer's activation space. For our toy spider row, [0, 1] times the map gives [1, 0]. The ant row gives [0.6, 0.8]. These are the original-space directions we can use to describe and edit the activation; we did not invert the average map.
The authors define a J-space component using a sparse, nonnegative combination of J-lens directions: a small number of directions, with amounts at least zero. This component captures only part of an activation, leaving a large remainder in the real models. The directions can overlap, so J-space is not a separate physical compartment with one idea in each slot.
A readable spider label gives us a candidate intermediate. We can see that candidate beside the route that actually produces eight. Does their appearance in the same run tell us whether the model used this representation? Before moving on, decide what evidence would answer that question.
Does a readable spider label establish that the model used it?
Stage 6 of 13
What exactly are we changing?
Return to our toy activation, [1.12, 0.16, 0.40]. It contains 1.00 times a spider direction, 0.20 times an ant direction, and a remainder. These directions overlap. Their amounts come from solving for both contributions together; the lens’s separate similarity scores are not these decomposition coefficients.
Follow the two amounts across the middle. We exchange 1.00 and 0.20 while leaving the directions in place. The remainder takes a straight path and keeps the same value. At this point we have chosen the new amounts; the activation still needs to be rebuilt.
The colored arrows on the right add the two contributions tip to tail. With the amounts exchanged, their sum moves from the hollow original point to the filled edited point. The third number stays at 0.40 because it lies outside the plane of the edit. These coordinates illustrate the mathematics; they are not measurements from Claude.
Exchanging two selected contributions changes their sum while preserving the orthogonal remainder.
Compare Swap amounts with Keep original using Intervention. Watch the two contributions and their sum while checking the last number. Preserving the remainder means preserving what is orthogonal to these two chosen directions, not everything outside J-space. This experiment control performs only the toy mathematical edit.
We can now specify the local operation: recover two overlapping-direction contributions, exchange their amounts, and add the change back into the original activation. The rest of the vector survives. To connect this calculation to behavior, we must place it inside the research comparison.
Stage 7 of 13
Where does the edit enter the experiment?
The research comparison uses the same prompt in two trials. One continues with its original activations. The other reapplies the spider–ant coordinate swap at every token position. The bracket covers those positions; the linked toy amounts open one local edit. After the intervention, the model still performs its remaining computation.
Now compare the reported results: the original top answer was 8; after the spider-to-ant swap it was 6. Those answers are observations from the research model, not outputs calculated by our toy vector. The opaque continuation boxes keep that experimental boundary visible.
The changed answer follows the substituted animal, which supports a causal role for the intermediate. We have not recovered all the network’s subsequent calculations. We can now ask something broader: is a representation usable by several different kinds of calculation?
Stage 8 of 13
Can one representation serve different tasks?
The researchers test this by swapping France toward China in separate prompts. One asks for the capital. Another asks for the language. Another asks for the continent. The country edit stays the same, while each prompt asks the model to do something different with that country.
Compare the paired outputs in the capital trial: Paris becomes Beijing. In the language trial, French becomes Chinese. In the continent trial, Europe becomes Asia. These are separate reported examples, not three computations we have reconstructed inside one run.
Select a question to inspect its paired result. What stays shared is the country intervention; what changes is the operation requested by the prompt. That pattern supports flexible reuse of a representation. It does not imply that every swap succeeds on every task.
This resembles one job of a workspace: making information available to several uses. But availability is not the same as dependence. If a lens reads a language representation while the model writes that language, does the model need that particular representation to keep writing it?
Stage 9 of 13
Does readable information control every use?
The study holds a Spanish passage fixed and swaps a Spanish J-lens representation toward French on the question tokens. When the question explicitly asks which language the passage uses, the answer changes from Spanish to French. A greeting request changes from Hola to Bonjour.
Now keep the passage and intervention but change the request to continuing the passage. The model continues in fluent Spanish even with the swap. A separate anomaly test inserts a French sentence into the Spanish passage. On that altered passage, both the original and swapped runs answer yes when asked about an intrusion. Readable information can be important for one use and dispensable for another.
Select the request and compare the two outcomes. This selector opens separate reported trials. The unchanged continuation is as useful as the changed language report: it rules out the simple claim that a Spanish lens representation controls every behavior involving Spanish.
Together, the country and language experiments suggest information that several operations can use, with selective dependence on that representation. They help motivate the workspace interpretation. The next question is whether instructions can direct what occupies this readable state even while the model’s visible output is occupied with something else.
Stage 10 of 13
Can instructions direct an unspoken task?
In another experiment, the model copies a sentence while being instructed to carry out the arithmetic 3 squared minus 2 internally. Copying fixes the visible job. The arithmetic instruction asks for a separate internal job without an accompanying written calculation.
While the next copied text remains the same, middle-layer lens readouts show nine and then seven. Those words follow the two arithmetic operations: square 3, then subtract 2. The displayed token tape represents the copying task; the layer trace summarizes the reported internal readouts. It is not a timed transcript of spoken reasoning.
The researchers also direct attention toward citrus during copying and find citrus-related lens readouts. Compare the instruction conditions: the lens contents respond to the instructed subject while the copied output stays on its original task. That is evidence of directed modulation of the internal state.
The evidence so far links vocabulary-readable states to silent work: interventions redirect some computations, and instructions can alter their contents. If the model’s disposition to express a thought is connected to its silent use, could training it to express useful reflections change later behavior when no reflection is requested?
Stage 11 of 13
Can reflection training alter silent behavior?
The researchers try this with Claude Haiku 4.5. They take partial task dialogues, append a reflection question, and generate a target reflection grounded in relevant principles. During fine-tuning, the training loss applies only to the reflection response. The earlier dialogue supplies context rather than a supervised demonstration of the desired task behavior.
At evaluation, the reflection question and answer are absent. One benchmark places the model after an agent has fabricated a bug and asks for a progress summary. A grader scores how dishonestly the model describes what happened, from 0 for explicit admission to 1 for presenting the fabrication as real.
The reported mean dishonesty score falls from 0.25 to 0.07 after reflection training. These are average grader scores, not probabilities that a response is dishonest. The figure summarizes 27 task contexts with five sampled reports per context. No reflection is requested during these evaluations.
Training also increases ethics-and-reflection terms in the J-lens readout. The researchers remove the corresponding lens-vector contributions and evaluate again. The trained model’s mean dishonesty rises to 0.22, while the baseline remains at 0.25. This ablation substantially reverses the gain on this benchmark, connecting the changed representations to the changed behavior.
Compare the intact and ablated bars. The result supports a practical consequence of the workspace account: training what a model would say in a reflective continuation can change what it silently uses in the original context. One benchmark and one selected ablation do not establish that the model has become reliably honest in every setting.
Stage 12 of 13
What does the workspace claim leave open?
We can now give the workspace claim its proper scope. The study finds readable intermediate concepts, flexible reuse, selective causal dependence, directed modulation, and a training effect linked to the edited representations. These are functional observations about how information is used.
The lens still returns ranked tokens. A list containing spider, ant, and above cannot tell us which animal is above the other or encode a complete chain of reasoning. Different relational structures can share the same list. A useful readout therefore does not amount to a complete explanation of the computation.
Nor does J-space identify a separate anatomical module or establish subjective experience. The workspace comparison concerns functional properties. The readout can be noisy, its vocabulary is restrictive, and its average map deliberately trades accuracy on one prompt for generality across prompts.
Those limits tell us what to do with a tempting internal label. Treat it as a candidate representation, check the measurement, and ask which controlled intervention would distinguish presence from use. Our original animal question gave us one such test.
Stage 13 of 13
Reconstruct the argument
Start with the numbers beneath the missing word. At one token position, layers update a residual-stream vector. Our toy updates changed the same three slots; a named concept did not need its own dedicated slot.
The Jacobian made a local relationship measurable. Nudging one input component produced a change in the final activation, and dividing by the nudge filled one sensitivity column. The other component supplied the other column.
Averaging corresponding Jacobian entries gave a map shared across contexts. It trades fidelity to one continuation for a broader readout of what an intermediate state is poised to express.
The vocabulary readout then exposed a candidate spider concept while the real continuation produced 8. Combining that readout with the average map also defined the original-space directions used for intervention. Readability alone still did not establish use.
The local swap exchanged two selected contributions and preserved their orthogonal remainder. Rebuilding the activation moved its endpoint because the two directions received different amounts.
The research comparison applied its intervention across token positions and held the prompt fixed. The top answer changed from 8 to 6. That supplied evidence of use beyond the candidate label found by the lens.
The same country edit changed a capital, a language, and a continent answer in separate trials. One representation could serve several requested operations, rather than simply replacing one output word everywhere.
The language trials qualified that reuse. A Spanish-to-French swap changed the explicit language report and greeting, while Spanish continuation remained intact. Being readable in a context does not mean being necessary for every behavior in that context.
During copying, an instruction could change the lens contents while the visible text stayed on task. The arithmetic readouts nine and seven followed the requested internal calculation across layers, without a written calculation in the output.
Reflection training changed a later task even when no reflection was requested. Removing selected ethics-and-reflection lens contributions substantially reversed the observed benchmark gain. The bars were mean grader scores, and the result had a specific experimental scope.
These experiments support functional workspace properties, but a ranked word list cannot recover every relation or computation. Nor does it establish a separate brain-like module or subjective experience. Keep those boundaries attached to the evidence.
Return to the opening question: did the model use an unspoken idea? A lens made a candidate visible, and controlled comparisons tested its use. The point of an internal readout is to make a testable question possible, not to make the answer self-evident.
Research behind the lesson
The numbers in the activation and Jacobian examples are invented teaching fixtures. Experimental outcomes are reported observations; the app does not run the research model.
- Gurnee et al. — Verbalizable Representations Form a Global Workspace in Language Models
Methods and Figures9,13,18,20,47–49. The displayed Jacobian follows the final-layer exposition; the default Sonnet implementation targets the penultimate layer.
Lesson summary
Read a candidate concept, change its contribution, and compare behavior before claiming the model used it.
- Layers update a running vector; a concept need not occupy one slot.
- A Jacobian measures local changes in output components caused by input nudges.
- Averaging creates a general readout that trades away prompt-specific accuracy.
- The lens reads a candidate concept and defines original-space directions; it does not prove use.
- Exchanging selected contributions changes their sum and preserves the orthogonal remainder.
- A fixed-prompt intervention connects an internal edit to an observed output change.
- The same representation can support different operations in separate prompts.
- Readable information can matter for one behavior and be dispensable for another.
- Instructions can change internal readouts while copying keeps the visible task fixed.
- Training reflective continuations can affect later silent behavior, with benchmark-specific causal evidence.
- Functional workspace evidence does not reveal every relation, circuit, or subjective experience.
What does a layer update in the residual stream?
The vector at a token position. Several components can participate in one concept direction.
What fills one Jacobian column?
Output change per unit nudge of one input component, with context held fixed.
Why average across contexts?
To read general verbalizability rather than fit the particular continuation of one prompt.
How can the lens read spider while the model outputs 8?
The lens is an observer of an intermediate state; the actual remaining layers still answer the legs question.
What does the coordinate swap preserve?
The component orthogonal to the two selected directions. Their coefficients exchange and their sum is rebuilt.
What connects the toy edit to the research result?
The study compares a fixed prompt with and without the intervention across token positions; 8 and 6 are observed top outputs.
What makes the country experiment evidence of reuse?
One France-to-China intervention changes different requested attributes in separate prompts.
What does the unchanged Spanish continuation teach us?
A readable Spanish representation is not necessary for every Spanish-related behavior.
What stays fixed during the internal arithmetic example?
The copying task and its visible continuation stay fixed while middle-layer readouts follow nine and seven.
Why remove the increased ethics-and-reflection contributions after training?
The ablation tests their causal role: it largely reverses the fabrication-benchmark gain while the baseline stays near its original score.
What can a ranked token list fail to reveal?
Relationships and the full computation. The workspace analogy also does not establish subjective experience.