Skip to content

Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

RL to incentivize agentic reasoning and tool use in LLM judge systems

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

TIR-Judge is an end-to-end reinforcement learning framework that trains LLM-based judges to interleave natural language reasoning with Python code execution, enabling verifiable, computation-grounded evaluations. An 8B parameter TIR-Judge outperforms text-only reasoning judges by up to 7.7% (pairwise) and matches Claude-Opus-4 on listwise benchmarks. Remarkably, TIR-Judge-Zero—trained with zero distillation—matches or beats the distilled variant through self-bootstrapping RL alone.

Problem

Existing LLM judges rely purely on text-based reasoning, making them brittle on tasks requiring precise computation (e.g., counting words, verifying numeric outputs, checking code correctness). Prior attempts at tool-augmented judging integrate tools only at inference time, preventing joint optimization of reasoning and tool-use, and are narrowly scoped to specific task types rather than general-purpose judgment.

Method

TIR-Judge extends a judge model \(J_\theta\) with a sandboxed Python interpreter \(I\). At each step \(k\), the judge produces a reasoning step \(r_k\) and code \(c_k\); the interpreter returns output \(o_k = I(c_k)\); the trajectory \(s_k = s_{k-1} \oplus r_k \oplus c_k \oplus o_k\) is fed back iteratively until a final prediction is emitted. Training uses DAPO (a GRPO variant) with a composite reward:

\[R = \begin{cases} 1 & \text{if } R_t = 1 \wedge R_f = 1 \wedge R_c = 1 \\ 0.1 & \text{if } R_c = 1 \text{ but } R_t = 0 \vee R_f = 0 \\ 0 & \text{if } R_c = 0 \end{cases}\]

where \(R_c\) is correctness (label match), \(R_f\) is format compliance (structured tags), and \(R_t\) penalizes code errors or exceeding 3 tool calls. Sandbox outputs are masked during loss computation to prevent memorization. Two training variants exist: TIR-Judge-Distill cold-starts with ~10k SFT trajectories distilled from Gemini-2.5-Flash, then runs RL; TIR-Judge-Zero iterates RS→SFT→RL purely from the base model. Training data (~26k preference pairs) spans verifiable domains (math, coding, instruction-following) and non-verifiable domains (helpfulness, safety, dialogue), with a heuristic suppressing tool calls for non-verifiable prompts.

Key Contributions

  • First framework to jointly optimize reasoning and tool-use in LLM judges via end-to-end multi-turn RL.
  • Composite reward signal (\(R_c, R_f, R_t\)) that rewards correctness, structured output, and efficient tool invocation together.
  • TIR-Judge-Zero: iterative self-bootstrapping (RS→SFT→RL loop) requiring zero teacher distillation, with TIR-Judge-Zero 4B achieving a 1.2% gain over TIR-Judge-Distill 4B.
  • Unified support for pointwise, pairwise, and listwise judgment formats in a single trained model.
  • 8-gram decontamination of training data against all seven evaluation benchmarks.

Results

  • Pointwise: TIR-Judge-Distill 8B achieves avg. 71.0% on PPE Correctness (vs. best text-only baseline RM-R1-Deepseek-Distill-7B at 66.9%), a +6.4% gain; TIR-Judge-Zero 8B reaches 70.3%.
  • Pairwise: TIR-Judge-Zero 8B achieves 75.7% avg. on PPE Correctness (vs. Think-RM 7B at 64.5%), a +7.7% gain over the strongest comparable text-only judge.
  • Pairwise vs. larger models: TIR-Judge-Zero 8B surpasses RM-R1-Deepseek-Distill-32B (72.0% avg.) with 4Ă— fewer parameters.
  • Listwise (RewardBench 2): TIR-Judge-Zero 8B reaches 78.4%, approaching Claude-Opus-4's level (~96% of its performance per paper text).
  • IFBench (instruction-following judge): TIR-Judge-Zero 4B scores 65.9%, vs. 44.1% for Qwen3-4B-Tool (inference-time tool use only).
  • CodeJudgeBench: TIR-Judge-Zero 4B scores 61.5% vs. 18.0% for Qwen3-4B-Tool, showing largest gains on computation-heavy tasks.

Limitations

  • Tool use is restricted to a Python interpreter; no search, symbolic solvers, or other tool types are explored.
  • Maximum 3 tool calls per trajectory is a hard heuristic; complex multi-step verification tasks may be under-resourced.
  • Non-verifiable domains (safety, helpfulness) cannot benefit from code execution, requiring a heuristic format penalty to suppress unnecessary tool calls—this coupling may be brittle.
  • Evaluation against proprietary closed models (GPT-4o, Claude) is partially from original papers rather than controlled re-runs, introducing potential comparability issues.
  • The iterative RS→SFT→RL loop for TIR-Judge-Zero adds significant training complexity and compute (8Ă— H100s, multiple rounds).

Relevance to Agentic AI / LLM Agents

TIR-Judge directly operationalizes a core agentic primitive—tool-integrated reasoning with closed-loop feedback—applied to the model evaluation problem rather than task completion. This is highly relevant because scalable, accurate LLM judges are a prerequisite for agentic RLHF pipelines, best-of-N selection, and autonomous quality gating in multi-agent systems. The TIR-Judge-Zero finding—that a judge can self-improve via iterative RL with no human or teacher supervision—demonstrates that agentic self-bootstrapping extends to meta-level evaluation roles, not just task-solving. The reward design (suppressing tool calls on non-verifiable tasks) also contributes a practical technique for teaching agents when tool invocation is worth its cost.