How ARTIST trains LLMs for agentic reasoning and tool use with reinforcement learning
See how ARTIST interleaves reasoning and tools, scores complete rollouts, applies GRPO and masks tool outputs during policy training.
Lesson summary
ARTIST trains the decisions around tool use from complete outcomes while keeping environment observations out of the policy loss.
- ARTIST puts reasoning, tool calls and returned observations inside one continuing trajectory.
- A completed rollout earns answer, format and tool-execution rewards without labels for every intermediate step.
- GRPO compares several rollouts from one task and derives a relative learning signal without a critic model.
- Tool outputs stay in context but are excluded from the loss because the environment, not the policy, produced them.
- Tool access through prompting did not match a policy trained to decide when and how to use the tool.
- On multi-turn tasks, ARTIST made more correct calls while reaching task completion in fewer aggregate steps.
What three kinds of segment can alternate before the final answer?
The policy can generate internal reasoning, issue a tool query and then continue from the tool or environment output.
Which three reward components are used in the paper’s math setting?
The math setting combines answer correctness, structured format and the fraction of successful tool executions.
What replaces a separate critic in GRPO?
Rewards from a group of sampled responses provide a relative baseline for their advantages.
Why can tool output remain in context while being absent from the loss?
The model needs the observation for later reasoning, but those deterministic tokens were produced by the environment rather than chosen by the policy.
What did the prompt-plus-Python baseline reveal?
Tool availability alone gave limited gains. ARTIST was trained to integrate tool use into the reasoning strategy and performed better on every reported math benchmark.
What changed besides function-calling accuracy?
ARTIST produced more correct tool calls and used fewer aggregate steps to finish the evaluated tasks.
Stage 1 of 8
The Tool-Access Gap
Begin with one exact integral. A model can reason fluently and still lose a sign during symbolic manipulation. Giving it Python removes the arithmetic limit, but a new problem appears: the model must decide when to call the tool and how to use what comes back.
The policy is the model making those choices. One complete attempt is a rollout. A tool output is an observation produced outside the model, and a reward is the score assigned after the attempt finishes.
We will keep the integral as our thread. First it passes through reasoning, a Python query and an exact result. Then several complete attempts compete, the policy learns from their outcomes, and one careful mask keeps the update attached to choices the model actually made.
That distinction is the whole mystery. ARTIST does not merely bolt a calculator onto an LLM. It trains the surrounding decisions as part of a trajectory. Let us start at the moment reasoning turns into an external action and then resumes.
Stage 2 of 8
Interleaved Reasoning and Tools
Our rollout opens with the integral and a tentative plan. The policy is still generating ordinary tokens, so it can decompose the expression, recognize that exact symbolic work is risky and decide that an external computation would help.
The tool call is written into the same trajectory as the reasoning. ARTIST’s structured format distinguishes the model’s thought, the Python query, the returned output and the final answer, which lets the runtime execute the query and append its result.
The environment returns `-2 + E`. That text was not generated by the policy, yet it now sits inside the growing context. What should the model do with it?
What happens to the returned tool output?
The next reasoning segment reads the observation, recognizes that `E` is Euler’s number and turns the tool’s syntax into the answer `e − 2`. The call therefore changes the reasoning that follows instead of behaving like a detached lookup.
A harder task can repeat this cycle. The policy may reason, call a tool, inspect an error, revise the query and call again. The paper treats each of those model actions and environment responses as one multi-turn rollout.
We have gained a useful execution loop, but no training signal yet. The runtime can carry out whatever the current policy attempts. Next we need a way to prefer trajectories that finish correctly, stay parseable and make executable calls.
Stage 3 of 8
Scoring a Complete Trajectory
The finished integral rollout arrives with all its intermediate reasoning intact. ARTIST does not require a teacher to label the moment Python should have been called or prescribe the exact line of reasoning that should follow.
Instead, the math setup checks three completed outcomes. A correct final answer earns two points. Proper tags and ordering earn up to one format point, while the fraction of successful Python executions contributes up to one more.
Suppose the answer is correct but every Python query fails to execute. Does correctness alone still earn the maximum rollout score?
Can a correct answer with failed calls receive the maximum score?
Our worked rollout is correct, fully structured and executable, so its three meters sum to four. Another trajectory can receive partial credit for clean structure or working calls even when its final answer is wrong.
This is outcome supervision rather than a script for each step. It tells training which completed behaviors were useful without claiming every intermediate choice was uniquely correct. The next problem is turning several such scores into a stable policy update.
Stage 4 of 8
GRPO Compares Several Rollouts
One score cannot say whether a trajectory was unusually good for this task. ARTIST’s math training samples six rollouts from the old policy for each question, so different reasoning and tool-use strategies meet the same problem.
GRPO uses the group’s rewards to form a relative baseline instead of training a separate critic model. Attempts above their group receive positive advantage, while weaker attempts receive negative advantage.
Imagine that all six trajectories earn exactly the same reward. With no response above or below the group, what relative preference remains?
What signal remains when every rollout reward is equal?
With mixed scores, the six trajectories spread around the group mean and produce opposing advantages. The clipped GRPO objective can then raise the probability of stronger model choices and lower the probability of weaker ones without a value-function critic.
Switch the Outcome pattern between mixed and equal scores. Mixed rewards create positive and negative relative signals, while equal rewards make the policy update neutral.
The group comparison explains which trajectories receive support, but a trajectory contains two authors. The policy wrote the reasoning and tool query, while the environment wrote the output. Next we must keep that ownership boundary inside the loss.
Stage 5 of 8
Masking Environment Tokens
Toggle the mask on the amber tool-output tokens. They always remain in the trajectory for later reasoning, but only the unmasked state sends environment text into the policy loss. Which update teaches the model’s own decisions?
Masking environment-produced output tokens keeps the policy update focused on the model’s reasoning and tool choices.
Stage 6 of 8
Tool Access Is Not Tool Strategy
Now we can test the original mystery. The paper evaluates the same 7B base model in three conditions: no added tool strategy, Python supplied through prompting, and ARTIST after reinforcement learning over agentic rollouts.
On AIME, Pass@1 rises from `0.040` for the base model to `0.122` with prompted Python access. ARTIST reaches `0.156`, so access helps but does not match a policy trained to integrate the tool.
If ARTIST had learned only to call Python more often, an easier benchmark should receive more calls too. Did the reported behavior follow that simple rule?
Did ARTIST simply increase tool calls on every math benchmark?
The measured pattern is selective. ARTIST averages over three tool calls on AIME compared with `0.3` for the prompt baseline, yet uses fewer calls on MATH-500, where internal reasoning is often sufficient. The improvement is strategy, not a universal call-more rule.
Switch the Math benchmark through all four choices. Compare the base, prompted-tool and ARTIST bars, then read the exact gain over the base model for each task.
ARTIST beats the prompted-tool condition on all four reported math benchmarks, though the size of the gain varies by benchmark. That supports the training claim in one domain. The paper also asks whether the same idea survives longer workflows with many functions and changing state.
Stage 7 of 8
Longer Tool Workflows
A travel or retail task is harder than one calculator call. The agent must choose several functions, carry identifiers and preferences forward, respond to failures and stop only when the environment reaches the intended state.
Across the paper’s τ-bench evaluation, the prompted-reasoning baseline makes `517` correct tool calls. ARTIST makes `676`. The extra correct actions matter because each one can unlock the next state transition in a longer workflow.
More correct calls could still mean the agent wandered through a much longer route. What happens to the total number of steps?
How does ARTIST’s aggregate step count change?
The two measures move in the useful directions together: `159` more correct calls and `236` fewer aggregate steps. Airline Pass@1 also rises from `0.120` for prompted reasoning to `0.260` for ARTIST.
The evidence does not settle safety or reliability in every open environment, which the paper leaves for future work. It does show one coherent mechanism: train complete tool-using trajectories, compare their outcomes and update only the choices owned by the policy.
Stage 8 of 8
The ARTIST Training Loop
We began with one integral and saw reasoning, a Python query and the returned observation become one continuing trajectory. The observation changed what the policy reasoned next.
The completed attempt then earned separate checks for its answer, structured format and executable calls. No teacher had to prescribe the one correct intermediate path.
Six math rollouts faced the same question, and GRPO compared their outcomes inside the group. Relative advantages supplied the learning signal without a separate critic.
The loss mask preserved a crucial ownership line. Tool outputs stayed available as context, but gradients passed only through tokens generated by the model.
The math comparisons showed why prompting was not enough. ARTIST learned to call tools heavily on AIME and more selectively on MATH-500 while improving accuracy.
The function-calling results carried the same idea into longer workflows. ARTIST made more correct calls and finished the evaluated tasks in fewer aggregate steps.
ARTIST’s central move is now visible as one loop. Let the policy act with tools, judge the completed outcome, compare alternative trajectories, and reinforce only the model-owned decisions that produced the stronger result.