Skip to content

Iterative Tool Usage Exploration for Multimodal Agents via Step-wise Preference Tuning

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SPORT is an iterative self-exploration framework that trains multimodal agents to use external tools without any human-annotated data. It generates synthetic tasks, samples candidate actions at each step, selects preferred actions via an AI verifier, and applies step-wise DPO to refine the VLM controller across iterations. On GTA and GAIA benchmarks it outperforms the SFT baseline (MAT-Qwen2-VL-7B) by 6.41% and 3.64% AnsAcc respectively.

Problem

Existing multimodal agent training—both SFT (distillation from GPT-4o) and RL—requires expensive human-annotated task–answer pairs and full tool trajectories. For complex multimodal tasks involving multi-format files, long reasoning chains, and diverse tool calls, such annotation is prohibitively costly and introduces distribution mismatch with target environments. Prior step-wise preference methods assume access to correct trajectories to construct preference pairs, which is unavailable here.

Method

SPORT operates in iterative rounds, each with four components:

  1. Task Synthesis: An LLM (Qwen2.5-7B) generates query–file pairs from seed examples; images are retrieved by similarity search, while DOCX/PDF/XLSX files are generated via LLM-produced Python code.
  2. Step Sampling: At each step i, the controller (Qwen2-VL-7B fine-tuned with LoRA) samples n parallel candidate actions (thought + code), executes each tool call to obtain observations.
  3. Step Verification: A multimodal LLM verifier, given the task context plus all n actions and observations, selects the best action. The best action becomes the preferred sample; the remaining n−1 form the dispreferred set, yielding m(n−1) step-level preference pairs per task (where m = trajectory length).
  4. Preference Tuning: Standard DPO is applied to the controller using the collected step-wise preference dataset D, with the post-SFT model as the reference policy π_ref.

The online search scheme alternates step sampling and step verification so that each step is conditioned on the best-so-far history, avoiding the compounding errors of trajectory-level evaluation. After tuning, the updated controller seeds the next iteration. Training uses LoRA (rank 32 on Q/K/V projections), AdamW + cosine scheduler, lr=1e-6, on 8×A100 GPUs; context window is 10,240 tokens.

Key Contributions

  • Annotation-free iterative exploration framework for multimodal tool-using agents (no ground-truth answers or trajectories required).
  • Novel online step sampling + AI-verifier scheme that extracts step-level preference pairs even from failed trajectories, improving data utilization.
  • A released 16K step-wise preference dataset collected via self-exploration.
  • Empirical demonstration that step-wise DPO substantially outperforms trajectory-level SFT and naïve static DPO baselines.

Results

  • GTA benchmark (229 tasks, 252 images): SPORT-Qwen2-VL-7B achieves AnsAcc 60.26%, ToolAcc 72.41%, CodeExec 91.87% vs. MAT-Qwen2-VL-7B (53.85% / 64.63% / 84.32%); +6.41% AnsAcc over SFT baseline.
  • GAIA benchmark (446 tasks across 3 levels): SPORT achieves 20.61% overall AnsAcc vs. MAT-Qwen2-VL-7B 16.97%; best among open-source model agents; +11% over HF-Agent with Qwen2-VL-7B.
  • SPORT outperforms closed-source-controller Lego Agent (GPT-4; GTA AnsAcc 30.20%) and matches or exceeds HF-Agent with GPT-4o (33.40%) on GAIA.
  • Static DPO baseline (MAT-SFT-DPO) gains only +1.28% AnsAcc over MAT-SFT; SPORT gains +6.41%, confirming the value of online exploration.
  • Ablation: iteration step size d=500 (2 iterations) outperforms d=200 (5 iterations) and d=1000 (1 iteration) across all metrics on GTA.
  • Verifier produces preferred/dispreferred pairs with 45.62% tool-distribution divergence; preferred steps have 18.35% code error rate vs. 81.94% for rejected steps.
  • Human study (20 researchers): 82% agreement between verifier preferences and human preferences; task/trajectory quality scores all exceed 8/10.

Limitations

  • The verifier relies on hand-crafted rules and prompts, limiting generalization to out-of-distribution tasks and edge cases.
  • No self-improvement loop for the verifier itself; verifier quality is a fixed upper bound on preference signal quality.
  • Performance still lags behind closed-source large-model agents (GPT-4o controllers), attributed to smaller model scale and less pre-training data.
  • Task diversity is a factor: reducing seed pool from 425 to 100 seeds drops AnsAcc from 60.26% to 58.33%, indicating sensitivity to synthesis breadth.

Relevance to Vision-Language Models

SPORT directly advances VLM-as-controller architectures by showing that a 7B-parameter VLM (Qwen2-VL) can be self-improved for tool use without human supervision, addressing a core bottleneck in scaling VLM-based agents. The step-wise DPO approach is a practical alternative to trajectory-level RLHF for multimodal settings where reward signals are sparse and costly to obtain. The online exploration scheme—where the VLM both generates candidate actions and informs preference selection—is a general recipe applicable to any tool-augmented VLM agent pipeline. This connects directly to the broader agenda of grounded, tool-using VLMs capable of handling real-world heterogeneous file formats and multi-step reasoning.