Skip to content

TACO: Tool-Augmented Credit Optimization for Agentic Tool Use

🕒 Published (v1): 2026-06-29 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

TACO is a GRPO variant for code-tool visual agents that assigns per-call credit through two coupled channels: Differential Answer-Probe Reward (DAPR), a self-supervised judge-free signal comparing the agent's answer before vs. after a tool call; and Outcome-Gated Advantage Routing (OGAR), which routes the final-answer advantage only to outcome-responsible token segments. Trained via a two-stage SFT+RL recipe, TACO achieves 68.1% average accuracy across 12 benchmarks (+4.4 over the best prior 7B agent) at the lowest end-to-end latency, learning to invoke tools only when they help.

Problem

Code operations in visual agents can be useful, redundant, or misleading — the same crop that turns a wrong answer right can also flip a correct answer wrong. Outcome-only rewards (standard RLVR) attach to the whole trajectory and cannot isolate a tool call's contribution; recent analysis (MED) shows that apparent crop-zoom RL gains are largely confounded by the model's own improvement. Existing process rewards either fail to attribute final correctness to individual tool calls (step-wise text rewards) or require an external judge model (e.g., GPT-4o in CodeV), adding API cost and inheriting its biases. No prior method provides a self-supervised, judge-free signal that scores a tool call by its own effect on the final answer.

Method

Differential Answer-Probe Reward (DAPR). Two lightweight probe decodes are inserted around the tool call: a pre-tool probe (tool-off answer \(a_1\): what the agent would answer without the tool) and a post-tool probe (tool-on answer \(a_2\)). Both are scored by the existing rule-based answer checker \(r_{\text{out}} \in \{-1,0,+1\}\). The tool value is the difference \(\Delta = r_{\text{out}}(a_2) - r_{\text{out}}(a_1)\): positive for a useful call, negative for a misleading one, zero when the answer is unchanged. Because \(a_1\) and \(a_2\) share the same question, image, and pre-tool reasoning, differencing cancels what the model "already knew," directly targeting the tool-gain confound. The same cancellation makes DAPR naturally robust to probe-hacking (pre-writing the answer into Think inflates both \(r_{\text{out}}(a_1)\) and \(r_{\text{out}}(a_2)\), leaving \(\Delta\) unchanged).

Outcome-Gated Advantage Routing (OGAR). A parameter-free rule that routes the final-answer advantage \(A_1\) to token segments based on outcome: - Useful (\(\Delta > 0\)): entire trajectory receives \(A_1\) credit. - Misleading (\(\Delta < 0\)): \(A_1\) blame lands on the tool branch (code + \(T_2\)) only, sparing the correct pre-tool reasoning \(T_1\). - Right-but-redundant (\(\Delta = 0\), \(a_1\) already correct): \(A_1\) withheld from the tool branch; credit stays with \(T_1\). - Necessary-but-failed (\(\Delta = 0\), \(a_1\) wrong): \(A_1\) blame withheld from the tool branch, encouraging exploration on hard items.

A process-channel advantage \(A_2\) (normalized \(\Delta\) over the group) acts on the whole sequence when \(\Delta \geq 0\) and is gated off (via \(g\)) for misleading calls. The combined loss is \(L_{\text{TACO}} = \alpha_1 L_{\text{GRPO}}(m \odot A_1) + \alpha_2 L_{\text{GRPO}}(g A_2)\), with \(m\) the per-token outcome gate, \(g\) the per-trajectory process gate.

Training. Two stages: (1) SFT cold-start on re-curated Thyme trajectories (execution-validated, tool-necessary, quality-filtered by Gemini-3-Pro); (2) GRPO with the gated dual-channel advantage, \(G=8\) rollouts, \(\beta=0\), on a single 8×80GB A100 node.

Key Contributions

  • TACO framework coupling DAPR and OGAR into a single GRPO-style objective for code-tool visual agents.
  • DAPR: a self-supervised, judge-free reward that scores each tool call by the before/after outcome difference — positive for useful, negative for misleading, zero for inconsequential — at zero API cost.
  • OGAR: a parameter-free, outcome-conditioned gate that routes the final-answer advantage only to responsible segments, suppressing wasted tool calls without a tool-cost penalty term.
  • Probe-hacking diagnosis and defense: identifies the vulnerability of generative probes to pre-written answers and shows that the difference-based design resists it, verified in training reward dynamics.

Results

  • Main accuracy (12 benchmarks, macro average): TACO-7B reaches 68.1%, surpassing the best prior code-tool agent (PyVision-7B, 63.7) by +4.4 and improving over CodeV-RL-7B (62.5), DeepEyes-v2-7B (61.2), and Thyme-7B (60.8) by 5.6–8.1 points. Outperforms GPT-4o (58.5) by +9.6.
  • Perception: leads all code-tool agents on HR-Bench-4K/8K, MME-RW, and \(\text{V}^*\) (e.g., \(\text{V}^*\): 89.6 vs. next-best 88.7).
  • Reasoning: strongest on fine-grained tasks (LogicVista 55.6, WeMath 53.1).
  • Latency vs. accuracy (Table 2): simultaneously the most accurate and fastest — e.g., \(\text{V}^*\): 89.6% at 2.3s/query vs. PyVision 88.7% at 3.6s.
  • Ablations (Table 3): full TACO 72.0% avg. on 5-benchmark subset; removing DAPR drops to 67.5 (use-sum-of-probes instead of difference); removing OGAR drops to 70.0.
  • Generalization (Table 4): TACO on Qwen2.5-VL-7B goes from 60.4 to 72.0 avg.; on Qwen3-VL-8B from 72.9 to 78.8.
  • Training dynamics (Figure 3): accuracy reward stays highest; completion length steadily decreases (fewer unnecessary tool calls); policy entropy remains healthy, unlike the additive-probe variant which exhibits probe-hacking (fast early rise then plateau).

Limitations

  • Evaluated only at 7B–8B scale; generalization to larger models (>30B) or alternative base architectures is not tested.
  • Requires a two-stage SFT cold-start with heavily curated trajectories (execution validity, tool necessity, Gemini-3-Pro quality filter) — the RL stage may not converge from an untuned base model.
  • DAPR's two probe decodes, while cheap, add latency overhead vs. pure outcome-only RL (though total latency remains lowest among agents due to fewer tool rounds).
  • Limited to single-turn visual code-tool agents with a single tool call per trajectory; multi-call chains and general multi-turn agentic scenarios are not addressed.
  • The rule-based answer checker constrains applicability to tasks with verifiable ground-truth answers; free-form generation is not supported.

Relevance to Agentic AI / LLM Agents

TACO tackles the fundamental credit-assignment problem for tool-use agents: how to determine, from outcome rewards alone, whether a specific tool call helped, hurt, or was irrelevant. Its self-supervised differencing approach (comparing the agent's answer with vs. without the tool) offers a general principle that extends beyond visual agents to any agentic system where a tool call creates a clean before/after boundary in the trajectory. By learning to suppress unnecessary calls, TACO directly addresses the over-calling tendency observed in outcome-only RL-trained agents, a key efficiency bottleneck for deployed LLM agents.