THOR: Tool-Integrated Hierarchical Optimization via RL for Mathematical Reasoning

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Hierarchical RL with tool integration for mathematical reasoning agents; strong results

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

THOR is a framework for tool-integrated reasoning (TIR) in LLMs that combines a multi-agent data synthesis pipeline (TIRGen), hierarchical RL training (episode-level + step-level), and a self-correction inference mechanism. It addresses the core failure modes of existing TIR methods: data style mismatch, sparse rewards in long chains, and ignoring tool feedback at inference. THOR achieves SOTA on mathematical benchmarks at comparable model scales.

Problem

LLMs struggle with high-precision mathematical tasks (numerical computation, symbolic manipulation) that require exact execution rather than probabilistic generation. Tool-Integrated Reasoning (TIR) is a natural fix, but three barriers block existing methods: (1) synthesized TIR data is misaligned with policy models (especially reasoning models like DeepSeek-R1); (2) episode-level RL suffers severe sparse rewards because long TIR trajectories make intermediate failures invisible to the training signal; (3) inference ignores immediate execution feedback, treating failed tool calls as dead ends rather than correction opportunities.

Method

TIRGen (data pipeline): A generator–refiner two-agent system. The Generator (\(\pi_{gen}\)) produces natural-language reasoning steps step-by-step up to a max length \(L_{step}\). The Refiner (\(\pi_{refiner}\)) judges each step for code-solvability, extracts the pure-logic portion, converts the calculation into executable Python, runs it in a sandbox, and splices the result back. Because the Refiner never sees the full problem or answer—only isolated steps—the resulting data stays in-distribution with the Generator's policy. A multi-stage filter removes format errors, trivially-coded examples, and problems solvable by pure chain-of-thought (CoT).

Hierarchical RL training (two-stage after cold-start SFT): - Episode-level: GRPO over full trajectories using binary final-answer correctness as reward \(R_i \in \{0,1\}\). Trajectories with failed code executions are filtered to avoid spurious penalties. An NLL auxiliary loss \(L_{NLL}\) reinforces high-advantage samples. - Step-level: Failed tool-call steps \(a^t \in A_{err}\) are extracted. Each erroneous thought \(r^t\) is split into a kept prefix \(r^t_{pre}\) and discarded suffix \(r^t_{suf}\); the model regenerates suffix + action from the prefix, receiving binary code-execution-pass reward \(r' \in \{0,1\}\). Advantage is normalized within the group. The combined objective is \(L(\theta) = L^{epis}_{\pi_\theta}(\theta) + L^{step}_{\pi_\theta}(\theta)\).

Self-correction at inference: When an action fails, the model backtracks to \(r^t_{pre}\) and regenerates only the suffix and action (up to \(N_{corr}\) attempts), adding negligible cost relative to full trajectory regeneration.

Self-rewarded BoN: At test time, \(N\) independent candidates are sampled without self-correction; the candidate with the highest code execution pass rate is selected—no external ORM required.

Key Contributions

• TIRGen: Policy-aligned TIR dataset construction via generator–refiner multi-agent pipeline, applicable to both reasoning and non-reasoning base models.
• Hierarchical RL: Joint episode-level (final-answer accuracy) and step-level (code execution pass) optimization with GRPO, motivated by the empirical finding that intermediate code-pass rate strongly predicts final answer correctness.
• Self-correction inference: Online backtracking on failed tool calls that regenerates only the erroneous suffix, not the full trajectory.
• Self-rewarded BoN: ORM-free best-of-N selection using execution pass rate as an intrinsic signal.
• Broad generalization: Single framework works across non-reasoning (Qwen2.5-Math-7B → THOR-7B) and reasoning (Qwen3-8B → THOR-Thinking-8B) models, including 1.5B/1.7B lightweight variants.

Results

• THOR-7B (non-reasoning, standard-scale) achieves 87.5 on MATH500, 50.0 on AIME 2024, 33.3 on AIME 2025, 81.3 on AMC 2023, avg 61.2—beating TORL-7B (avg ~35.7), TATA-7B (~24.7), and approaching or surpassing GPT-4o-1120 on several benchmarks.
• THOR-Thinking-8B (reasoning, standard-scale): 96.8 MATH500, 77.5 AIME 2024, 62.5 AIME 2025, 96.8 AMC 2023, avg 79.8—outperforming START-32B (avg 67.5) and QwQ-32B-Preview (avg 59.7) with 4× fewer parameters.
• THOR-1.5B (non-reasoning lightweight): avg 49.1 vs Qwen2.5-Math-1.5B at 18.4.
• THOR-Thinking-1.7B (reasoning lightweight): avg 65.3 vs Qwen3-1.7B at 61.1.
• Self-rewarded BoN (\(N=8\)): +2.1% avg over THOR-7B single-path; +3.4% over THOR-Thinking-8B; combining with external RM yields +3.2% / +4.5%.
• Code benchmarks (HumanEval+, MBPP+, LiveCodeBenchv6) also improve consistently.

Limitations

• Step-level optimization dataset \(D_{step}\) is constructed from execution-failed steps within sampled trajectories; if the base model rarely produces failed steps (highly capable models), this signal is scarce.
• Context length is capped at 4,096 tokens for non-reasoning models and 16,384 for reasoning models, which may truncate long TIR trajectories on harder problems.
• Self-correction adds up to \(N_{corr}\) regeneration attempts per failed step; while lightweight compared to full trajectory regeneration, this overhead accumulates for problems with many failures.
• TIRGen's Refiner must itself be capable of code generation and instruction following; the quality ceiling on generated data is bounded by the Refiner's ability.
• Evaluation uses LLM-as-Judge (Qwen3-32B), introducing potential judge bias or inconsistency.

Relevance to Agentic AI / LLM Agents

THOR directly instantiates the think–act–observe agentic loop: the model maintains a trajectory \(\tau = (r^1, a^1, o^1, \ldots, r^n)\) and uses tool feedback dynamically—a standard design pattern for LLM agents using code interpreters or external APIs. The hierarchical RL contribution is especially relevant: it shows that step-level dense rewards derived from intermediate tool-call success can substantially alleviate the sparse-reward problem that plagues long-horizon agent training, a challenge central to RL-from-environment-feedback approaches like those used in WebArena or SWE-bench agents. TIRGen's generator–refiner multi-agent data construction pipeline also demonstrates a practical recipe for bootstrapping policy-aligned agentic training data without relying on a stronger external model to label every step. The self-rewarded BoN (using tool execution pass rate as an intrinsic signal) extends to any agentic setting where intermediate actions produce verifiable outcomes, making it a broadly applicable inference-time scaling strategy.