How ATLAS trains small language-model agents for large MCP toolspaces
See how ATLAS combines iterative tool loading, programmatic orchestration and rubric-based RFT for efficient MCP agents.
Lesson summary
ATLAS scales small-agent capability by learning what context to acquire, keeping execution state compact and assigning credit with fixed...
- ATLAS reveals server names, tool names and full schemas in progressively richer layers.
- Loading only task-relevant schemas preserves context for the work that still has to happen.
- Programmatic orchestration keeps intermediate results in variables instead of replaying them through the prompt.
- Selective loading helps efficiency, but the paper’s cold-start 4B result does not improve until execution or learning is added.
- Fixed task rubrics split fuzzy success into comparable checks that a smaller judge can score.
- The paper reports 4.15 TF for the full 4B system versus 4.38 for the 1T frontier baseline.
- The headline comes from synthetic tasks and LLM judging, with several internal reporting tensions left unresolved.
What are the three information layers in ATLAS tool discovery?
The agent begins with a server index, opens a compact list of tool names for one server and then loads full details only for a selected tool.
What does iterative tool loading materialize on demand?
It materializes the detailed schema and examples for selected tools while leaving unneeded tools as compact names.
Where do intermediate tool outputs live during programmatic orchestration?
They remain in Python program state as native values instead of being repeatedly injected into the model prompt.
What happened when the cold-start 4B agent moved from ISL to ITL?
Average tokens fell slightly from 9,152 to 9,045, while reported task fulfillment fell from 2.73 to 2.36.
Which four categories organize the paper’s task rubrics?
They are task fulfillment, tool appropriateness, grounding in tool outputs and parameter accuracy.
What are the two headline task-fulfillment scores?
The paper reports 4.15 for the full Qwen3-4B configuration and 4.38 for the Kimi-K2 Thinking frontier baseline.
Why is 4.15 not a literal 41.5 percent task-completion rate?
It is an LLM-judged task-fulfillment score on a 0-10 scale over synthetic but realistic benchmark tasks.
Stage 1 of 9
The Context Squeeze
A user asks a 4B model to find a national park, compare tomorrow’s forecasts and convert the winning temperature to Celsius. The request needs three servers, but the model is connected to a much larger tool ecosystem. If every verbose schema arrives first, the task starts inside a crowded prompt.
An MCP server groups related capabilities, and each tool schema explains a callable name, arguments and structure. Context is the model’s limited working input. A trajectory is the whole attempt: decisions, calls, returned evidence and the final answer.
The obvious repair is to load less, but that creates a new decision: the model must discover the right tools without seeing every detail. Then it must preserve state across many calls and learn from tasks whose success cannot be checked by one exact answer.
ATLAS joins three answers to those three pressures. It loads context in stages, stores long-horizon execution in code and trains the resulting behavior with fixed task rubrics. Let us begin with the path from a compact server index to one callable schema.
Stage 2 of 9
From Server Index to One Full Schema
The agent needs a practical way to reach a selective tool context without already knowing every tool. Iterative Server Loading begins with a compact index of available servers. The task points toward National Parks first, so only that cabinet opens.
A server can expose many tools, so opening it still does not justify loading every schema. Iterative Tool Loading initially reveals only compact names. That is enough for the model to sketch a plan while preserving context for execution.
The plan now chooses park search. Precision matters because the call needs the correct arguments, but the other tool names remain irrelevant. Does ATLAS expand every tool in the server or only the selected one?
Which details are materialized after one tool is selected?
Only the chosen tool expands into a full callable signature and example. After the park list returns, the same pattern can open Weather Data and later Unit Converter. Detail arrives just in time, not as a permanent tax paid before planning begins.
The loading path is now clear: server index, tool names, then one full schema. Next we can manipulate that policy directly and see the boundary between a context that is incomplete and one that is crowded.
Stage 3 of 9
Choose Which Schemas Enter Context
Load tool slips from the three MCP cabinets. The required search, forecast and conversion schemas make the workflow ready, while loading every available slip crowds out its working room. Once the context is ready, the next question is where the workflow should keep its growing intermediate state.
Stage 4 of 9
Keep Long-Horizon State in Code
Suppose park search returns several candidates. In a conventional JSON loop, that result is injected into the prompt, the model reasons again, then another full output returns after the weather call. Context grows because the model repeatedly carries the conversation and the data.
ATLAS instead mediates calls through a persistent Python interpreter. Normalized MCPServer functions make unfamiliar APIs look like callable Python, while loops and conditionals express the workflow directly.
The weather response may contain many records, but the final answer needs only the coolest park. Must every record come back through the prompt before the next call?
Where can the full forecast objects remain?
The program stores parks and forecasts as variables, selects the coolest record and passes only that value into unit conversion. If an argument is wrong, the scaffold returns a focused hint and the model edits the local program instead of replaying the entire reasoning trace.
Switch the Execution mode. JSON turns replay intermediate output into the prompt; Program keeps it in variables. Compare both outcomes.
The execution substrate is now disciplined, but a cold-start model does not automatically know how to use it. The paper’s measurements make that limitation unusually clear, so the next stage separates lower token use from higher task success.
Stage 5 of 9
Lower Context Is Not Yet Better Behavior
For Qwen3-4B, Iterative Server Loading averages 9,152 tokens and Iterative Tool Loading averages 9,045 in Table 1. The interface is leaner, although this particular measured difference is small.
Efficiency is only useful if the agent still finishes the task. On the paper’s 0-to-10 task-fulfillment scale, the same cold-start move goes from 2.73 under server loading to 2.36 under tool loading.
The model saw slightly fewer tokens, yet its task score fell. Does selective loading by itself solve the agentic problem?
What does the cold-start ITL result establish?
Adding programmatic orchestration lifts the cold-start ITL score from 2.36 to 2.94. That does not finish the story, but it provides a more stable execution substrate. The paper’s larger gains arrive when reinforcement finetuning teaches the decisions around that structure.
A training signal now has to distinguish a grounded multi-tool solution from a plausible answer that chose the wrong tool, misused a result or supplied a bad argument. One generic score hides those different failures, which is why ATLAS turns the task itself into a rubric.
Stage 6 of 9
Turn Fuzzy Success into Comparable Credit
Many MCP tasks allow several valid routes and final wordings, so exact-answer grading is too narrow. ATLAS generates a task-specific rubric once, before training evaluates any particular rollout.
Each criterion belongs to one of four categories: task fulfillment, tool appropriateness, grounding in tool outputs and parameter accuracy. The same written checks and weights stay fixed while the judge compares rollouts for that task.
A shortcut can sound convincing while ignoring the forecast returned by the weather tool. If both final answers are fluent, can the fixed rubric still separate them?
Which evidence lets the rubric reject the shortcut?
The fixed gauges expose the shortcut: it may look fluent, yet weak tool choice and missing grounding pull down its composite signal. The exact values are illustrative, but the structure matches the paper’s weighted category reward.
Switch the Trajectory. The same four criteria expose the shortcut and reward the grounded path. Then solve the challenge.
Concrete criteria also reduce the burden on the judge. In the paper’s 4B ISL comparison, the rubric-guided Qwen3-30B judge reaches 3.87 task fulfillment, above the 3.43 result from a rubric-guided GPT-4o judge. Next we will place the complete reported system beside the frontier baseline.
Stage 7 of 9
The Reported 4B-to-Frontier Result
Begin at the cold-start ITL score of 2.36. Adding programmatic orchestration moves that variant to 2.94, while generic reinforcement finetuning without PTC reaches 3.48. Each row tests a different combination rather than one smooth training run.
Combining PTC with generic RFT reaches 3.91. The paper argues that structured execution gives learning a more stable long-horizon substrate, while reinforcement supplies the largest improvement in behavior.
The frontier reference is Kimi-K2 Thinking with all tools eagerly loaded. The paper lists it as a one-trillion-parameter model using an 80K context, and Table 1 reports 4.38 task fulfillment with 23,768 average tokens.
The final reported Qwen3 agent has four billion parameters, iterative tool loading, programmatic execution and rubric-based reinforcement finetuning. How close does its bar stop to 4.38?
Which task-fulfillment score does Table 1 report for the full 4B configuration?
The 4B bar reaches 4.15 and the frontier bar reaches 4.38 on the same reported scale. That is a 0.23-point gap, and 4.15 is 94.7 percent of 4.38. The smaller system also averages 13,400 tokens in this table versus 23,768 for the frontier baseline.
The paper reports a 4B ATLAS agent within 0.23 task-fulfillment points of its 1T frontier baseline.
This is the paper’s strongest result, not a claim that a 4B model has become generally equivalent to a frontier model. To understand what the comparison establishes, we need to follow the data, filters and judges that produced the score.
Stage 8 of 9
What the Benchmark Can Establish
The training pool begins with more than one thousand synthetic but realistic MCP tasks. Automated solvability and utility filters, followed by execution with a large Kimi model, leave 304 tasks spanning one to three servers.
Evaluation uses 104 held-out MCPBench tasks on the original 28 servers and 100 ATLAS-Test tasks that include 11 additional servers. This tests new tasks and new tool combinations, but it is still a constructed benchmark rather than an uncontrolled production environment.
Task fulfillment is an LLM-judged score from zero to ten, not a literal completion percentage. Model judges also help filter training data and generate or apply rubrics, so the result depends on that measurement pipeline.
The paper also contains tensions worth preserving. Table 1 and section 5.5 report the full 4B rubric-plus-PTC result, while section 5.3 says rubrics were not yet applied to PTC. A cold-start decline is called a gain, and filtering thresholds differ between the main text and appendix.
The cautious conclusion is still substantial: within these judged MCP benchmarks, learned context acquisition, programmatic execution and task rubrics produce large gains for small agents. Real-world reliability, cost and reconciliation of the paper’s internal reporting remain open work.
Stage 9 of 9
The Complete ATLAS Argument
We began with progressive discovery. The model saw a compact server index, compact tool names and full details only for the tool it was ready to call.
Loading individual tool slips then exposed the trade-off directly. The useful context was not the smallest possible prompt; it was the smallest prompt that still held every schema required to finish the task.
Programmatic orchestration moved long-horizon state out of repeated JSON turns. Python variables retained intermediate objects, and local program edits repaired errors without replaying the entire trace.
The cold-start comparison stopped us from confusing efficiency with competence. ITL used slightly fewer tokens but scored lower until stronger execution structure or reinforcement learning taught the new behavior.
Fixed task rubrics made weak supervision more precise. They separated fulfillment, tool choice, grounding and parameter accuracy so a smaller judge could score concrete criteria consistently.
The mechanisms culminated in the reported comparison: 4.15 task fulfillment for the full 4B configuration and 4.38 for the one-trillion-parameter frontier baseline.
Finally, we kept the measurement chain attached. These were synthetic MCP tasks scored by model judges, and the paper leaves both deployment questions and several internal reporting tensions unresolved.
The complete argument is now visible. ATLAS does not make a small model capable by giving it an ever larger prompt. It teaches the model what context to acquire, gives execution a compact stateful form and supplies task-aligned credit for learning those decisions.