Skip to content

InSight-o3: Empowering Multimodal Foundation Models with Generalized Visual Search

🕒 Published (v1): 2025-12-21 14:23 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

InSight-o3 is a two-agent multimodal framework that decouples "thinking with images" into a high-level reasoning agent (vReasoner) and a specialized visual search agent (vSearcher) trained via hybrid reinforcement learning. The vSearcher (InSight-o3-vS) operates as a plug-and-play sub-agent that locates free-form-described regions in arbitrary images, boosting frontier models like GPT-5-mini from 39.0% to 61.5% on the authors' new O3-Bench. A companion benchmark, O3-Bench (345 QA items over dense charts and maps), targets the joint perception+multi-hop reasoning gap that prior datasets miss.

Problem

Current open multimodal models are inadequate at tasks requiring interleaved visual perception and multi-step reasoning on high-information-density images (e.g., composite charts, venue maps). Existing visual search systems handle only simple object references in natural images, and prior benchmarks do not jointly stress-test multi-region evidence gathering alongside higher-order reasoning, leaving a large gap between open models and proprietary systems like OpenAI o3 (40.8% on O3-Bench).

Method

System architecture. InSight-o3 is a two-agent pipeline: the vReasoner (a strong MLLM, e.g., GPT-5-mini) decomposes questions and issues free-form region descriptions; the vSearcher (Qwen2.5-VL-7B-Instruct fine-tuned as InSight-o3-vS) localizes those regions via bounding-box prediction and image-cropping tools, returning cropped evidence.

Generalized visual search. Unlike prior work restricted to discrete object references in natural images, vSearcher handles relational, fuzzy, or conceptual descriptions (e.g., "the chart showing company revenue") on arbitrary image types.

Hybrid sub-agent RL training. - Out-of-loop RL: vSearcher is trained on pre-generated (image, free-form description, ground-truth bbox) triples from InfographicVQA. Reward combines a format signal and an IoU reward (thresholded at α; boxes below α yield no reward), conditioned on whether the model made at least one tool call. - In-loop RL: vReasoner generates visual search tasks on-the-fly during training. The reward uses a pseudo-IoU signal: vReasoner rates each vSearcher crop (binary helpfulness score s), gated by whether vReasoner ultimately answers correctly (c); reward = 1 only when s=c=1. - Advantage estimation follows GRPO for out-of-loop (group normalization) and global mean/std normalization for in-loop tasks (no natural grouping). - Training vReasoner is fixed; only vSearcher's weights are updated.

O3-Bench construction. 204 images (117 composite-chart, 87 digital-map), 345 six-choice QA items. Pipeline: layout detection (PP-DocLayout+) → GPT-5-generated questions (without full image, forcing region-level composition) → human filtering/authoring → difficulty filtering (discard items solved by all three frontier MLLMs: GPT-5-mini, Gemini-2.5-Flash, Doubao-Seed-1.6).

Key Contributions

  • O3-Bench: a 345-item benchmark requiring multi-hop reasoning over high-resolution, information-dense charts and maps; significantly harder than MME-RealWorld and perception-centric predecessors.
  • InSight-o3: a two-agent "divide-and-conquer" framework separating visual reasoning from visual search, enabling plug-and-play composition with arbitrary vReasoners.
  • InSight-o3-vS: a Qwen2.5-VL-7B-based vSearcher trained via hybrid in-loop/out-of-loop RL for generalized visual search over arbitrary images and free-form region descriptions.
  • Hybrid RL algorithm: combines IoU-supervised out-of-loop RL (fast, stable) with on-policy in-loop RL (aligned with inference distribution), outperforming either alone.
  • Cross-model generalization: vSearcher trained under GPT-5-mini transfers to GPT-5-nano, GPT-4o, and Gemini-2.5-Flash without retraining.

Results

  • GPT-5-mini + InSight-o3-vS vs. GPT-5-mini alone: 39.0% → 61.5% on O3-Bench (+22.5 pp); 53.7% → 64.9% average across six benchmarks (+11.2 pp).
  • Gemini-2.5-Flash (3500px) + InSight-o3-vS: 60.4% → 69.7% on O3-Bench; 61.7% → 63.7% average.
  • GPT-5-mini + InSight-o3-vS surpasses OpenAI o3 (40.8%) on O3-Bench (61.5%).
  • GPT-5-mini gap with Gemini-2.5-Flash on O3-Bench narrows from 21.4 pp to 8.2 pp with vSearcher.
  • Hybrid RL (in+out) outperforms out-of-loop only (+7.1 pp on O3-Bench for GPT-5-mini) and in-loop only (+2.7 pp), while being faster than in-loop only (941s vs. 693s vs. 1223s per step).
  • Untrained Qwen2.5-VL-7B as vSearcher generally hurts performance or yields marginal gains; InSight-o3-vS consistently improves it.
  • vSearcher is resolution-insensitive: at 0.8M pixels (25% of training resolution), performance degrades only slightly, with more search calls compensating.

Limitations

  • vReasoner is not jointly trained; the framework relies on a fixed, often proprietary model (GPT-5-mini) for high-level reasoning, creating a dependency on API access at training time.
  • On some benchmarks (VisualProbe-Hard with GPT-4o and Gemini-2.5-Flash), adding the vSearcher degrades performance, attributed to weak tool-calling and multi-turn reasoning in those vReasoners.
  • O3-Bench is small (345 items), which limits statistical power and may introduce evaluation variance.
  • vSearcher (Qwen2.5-VL-7B) averages only ~1 image-crop tool call per query despite encouragement, due to model reluctance and vReasoner issuing coarse descriptions that need no verification.
  • Training data for in-loop RL uses synthetic image collages that differ substantially from real evaluation distributions; the gap is mitigated but not eliminated.
  • Credit assignment across the two agents during in-loop RL remains imperfect; the proxy reward (vReasoner feedback + outcome) can be noisy.

Relevance to Harnesses / Meta-Harnesses

InSight-o3 is a concrete instantiation of a meta-harness pattern: a fixed orchestration layer (vReasoner as orchestrator) dynamically dispatches sub-agents (vSearcher) to collect evidence, then synthesizes results into a final answer—exactly the structure of a tool-augmented harness controlling specialized workers. The hybrid RL training procedure for vSearcher is itself a harness-aware training paradigm, where the sub-agent is optimized against reward signals generated by the orchestrating agent at runtime (in-loop) and pre-generated offline (out-of-loop), demonstrating how harness topology shapes training. The plug-and-play design principle—vSearcher generalizes across vReasoners without retraining—offers a model for building composable, swappable sub-agents within larger harness pipelines. For researchers tracking harnesses and meta-harnesses, the paper provides a worked example of reward design, credit assignment, and stability tradeoffs when jointly optimizing a two-agent system with distinct and interdependent objectives.