How Slowrun LLM experiments stack into research decisions
Trace Slowrun ideas through interventions, noisy measurements, cumulative records, cross-track evidence, interactions, and honest decisions.
The whole story in 5 lines
Slowrun decisions improve when recorded gains are separated from noise, inherited context, interactions, and unresolved uncertainty.
- A useful experiment starts with a claim that one controlled intervention can challenge.
- A smaller loss becomes evidence only after its size is judged against run-to-run variation.
- A cumulative record credits the whole recipe more safely than the newest ingredient alone.
- Agreement across compute tracks raises confidence, but different stacks still leave confounding.
- Matched combinations reveal whether useful changes add, amplify, or cancel each other.
One record, many possible causes
A new Slowrun recipe reaches a lower validation loss after several ideas were stacked. The score improved, but the result does not identify which idea deserves credit.
A hypothesis is a claim we can challenge. An intervention is the change we make, while noise is ordinary variation between otherwise similar training runs.
An interaction appears when two changes behave differently together than they did alone. That possibility makes a winning stack useful evidence, but not a clean explanation.
Our journey follows one question through measurement, cumulative records, cross-track comparison, and interaction testing. Now let us start with the experiment before the score arrives.
Turn an idea into a comparison
We begin with the fixed Slowrun benchmark: 100 million FineWeb tokens and validation loss as the score. Lower loss means the model predicts held-out text better. The shared benchmark keeps the research target stable.
The research claim must name a mechanism before training starts. For example, epoch shuffling might reduce repeated ordering patterns and improve generalization. Writing this claim early protects us from inventing a story after seeing the result.
Both runs should share data, budget, evaluation, and everything else we can hold fixed. Which branch should receive the proposed intervention?
The shared starting recipe now splits into baseline and treatment runs. Only the treatment receives the change, so their later difference can challenge the original hypothesis. Both branches still need the same evaluation procedure.
A controlled split gives the idea a fair test, but training is not perfectly repeatable. Next, we will ask whether the measured difference is larger than ordinary run noise.
Separate a loss drop from run noise
The Slowrun ledger records 3.402 for the limited baseline and 3.376 after epoch shuffling. That 0.026 drop is recorded, but one pair cannot reveal run variation.
To teach the uncertainty, we add clearly marked illustrative repeats around those recorded points. Their horizontal spread represents different seeds or other uncontrolled training variation.
A lower average can still overlap the baseline outcomes when variation is wide. Would the same observed gap support a decision under noisy repeats?
The repeated outcomes now expose the missing quantity: uncertainty around each measured loss. A gap matters only in relation to that spread, not by its decimal places alone.
Switch Repeatability between Noisy and Stable. Compare whether the treatment outcomes overlap the baseline cluster and whether the decision readout changes.
The published 0.026 loss drop is real, while the repeat clouds are illustrative because the public record lacks replicate distributions. Next, we will inspect what cumulative records can safely tell us.
Read a record as an inherited recipe
Now that measurement has limits, let us read the limited-track history carefully. The published sequence begins at 3.402 and descends as record recipes replace earlier ones.
Epoch shuffling reaches 3.376, then value projections reach 3.349. Each point is an exact recorded result, and each new recipe inherits work from the prior record.
SwiGLU, U-Net, and head gating continue the descent to 3.295. Does the final step prove that head gating alone caused the full 0.107 gain?
The marginal drops are useful clues, but the interventions were not all isolated against one fixed baseline. Later values mix the newest change with inherited settings and schedule differences.
The complete record path now shows two truths together. Slowrun made substantial progress, and causal credit becomes less certain as the stack grows deeper.
A record ladder is excellent for finding better recipes, but it is weaker for explaining each ingredient. Next, we will compare repeated directions across compute tracks.
Look for repeated direction across budgets
We just saw why one cumulative sequence leaves confounding. Slowrun also records a tiny track, which provides a second budget where several ideas appear again. This creates useful overlap without creating a controlled replication. It still deserves attention.
SwiGLU, U-Net, and head gating each accompany new records in both the limited and tiny histories. The exact loss scales differ, so we keep each track separate. Their model sizes and time limits differ too.
Repeated downward direction makes a one-track accident less plausible, but the starting recipes and budgets still differ. Is this equivalent to a matched replication?
The paired lanes now descend together for the three repeated techniques. That pattern raises confidence in direction, while the separate baselines preserve the remaining uncertainty.
Cross-track agreement is stronger than one isolated win, but it still cannot reveal whether two ingredients cooperate. Next, we will use a matched combination test to expose interaction.
Test changes alone and together
★ If you remember one thing · A combined result can depart sharply from the sum of two individual gains.
Cross-track agreement gave us a useful clue, but stacking asks a different question. We need baseline, A alone, B alone, and A plus B under matched conditions.
The first three cells estimate the two individual gains. From them, simple addition predicts where the combined cell should land if the changes act independently.
The combined run can land on the additive guide, below it, or above it. Which outcome would show that the changes erase part of each other?
Three matched grids now reveal additive, synergistic, and antagonistic combinations with the same encoding. This contrast is illustrative, because Slowrun does not publish this exact factorial set.
Switch Interaction through Additive, Synergy, and Conflict. Compare the combined corner with the dotted prediction and inspect how the consequence readout changes.
A matched 2x2 identifies interaction because every required corner is measured. Slowrun would need runs like these before assigning precise causal credit. Now let us step back and see the whole picture together.
Turn experiment history into a decision
We started with a hypothesis and one controlled intervention. That discipline made the result capable of challenging a specific claim.
Then we separated a recorded loss drop from unknown run variation. The public record supplies point estimates, but not replicate distributions.
Next, we read the limited-track history as accumulated recipes. The sequence proves progress more strongly than isolated causal credit.
We compared limited and tiny tracks next. Repeated downward direction supported transfer while different budgets and inherited stacks kept confounding alive.
Finally, the illustrative 2x2 showed how matched combinations identify interaction. We avoided a numeric Bayesian posterior because no prior or likelihood was supplied.
Together, these evidence types support a careful decision: keep promising changes, run matched confirmations, test combinations, and carry unresolved uncertainty into the next hypothesis.
The whole story in 5 lines
Slowrun decisions improve when recorded gains are separated from noise, inherited context, interactions, and unresolved uncertainty.
- A useful experiment starts with a claim that one controlled intervention can challenge.
- A smaller loss becomes evidence only after its size is judged against run-to-run variation.
- A cumulative record credits the whole recipe more safely than the newest ingredient alone.
- Agreement across compute tracks raises confidence, but different stacks still leave confounding.
- Matched combinations reveal whether useful changes add, amplify, or cancel each other.