InSight-o3: Empowering Multimodal Foundation Models with Generalized Visual Search¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

InSight-o3 is a two-agent multimodal framework that decouples high-level visual reasoning (vReasoner) from region localization (vSearcher), enabling "thinking with images" on dense, multi-hop tasks. It introduces O3-Bench, a new benchmark of 345 QA samples on high-resolution charts and maps that requires cross-region evidence aggregation, and InSight-o3-vS, a vSearcher model trained via hybrid reinforcement learning to localize relational/fuzzy/conceptual regions in free-form language. The plug-and-play vSearcher significantly lifts frontier proprietary models on both O3-Bench and established benchmarks.

Problem¶

Current multimodal agents conflate reasoning and perception into a single model, causing failures on tasks that require iterative, interleaved attention to multiple distinct image regions (e.g., cross-chart analysis, venue-level map navigation). Existing benchmarks are near-saturated and test either coarse perception or cognition-heavy STEM questions with visually simple images—none jointly stress multi-hop reasoning and multi-region fine-grained visual search on information-dense images. Even frontier systems like OpenAI o3 score only 40.8% on the new benchmark.

Method¶

InSight-o3 separates "thinking with images" into two roles:

vReasoner: a frontier MLLM (e.g., GPT-5-mini) that decomposes questions, issues free-form region descriptions as search requests, and integrates returned crops to produce a final answer.
vSearcher (InSight-o3-vS): a specialized MLLM trained to localize regions given arbitrary free-form descriptions (generalized visual search), returning bounding-box crops for vReasoner to consume.

Training uses a hybrid sub-agent RL algorithm combining: 1. Out-of-loop RL: pre-generated (image, description, GT bbox) triples from InfographicVQA; reward is \(r = \mathbb{I}[n_\text{tool}>0]\cdot(\lambda_\text{format}\cdot r_\text{format} + \lambda_\text{IoU}\cdot r_\text{IoU})\) where \(r_\text{IoU} = \max\{0, \mathrm{IoU}(b,b^*)-\alpha\}/(1-\alpha)\). 2. In-loop RL: vReasoner generates search tasks on-the-fly during training; vSearcher is rewarded via a pseudo-IoU \(\hat{r}_\text{IoU} = \mathbb{I}[s = c = 1]\) where \(s\) is vReasoner's helpfulness rating and \(c\) is final-answer correctness—outcome supervision guards against unreliable ratings. Advantages follow GRPO with global (not group) normalization since dynamic tasks have no fixed group.

Training data is synthesized: image collages (stitched multi-image canvases with a target QA image + distractor fillers) for in-loop RL, and GPT-5-nano-generated region descriptions over InfographicVQA layout boxes for out-of-loop RL.

Key Contributions¶

O3-Bench: 345 multi-choice QA samples over 204 high-resolution chart/map images, requiring cross-region evidence aggregation and multi-hop reasoning; filtered so no item is solved by all three tested proprietary models.
InSight-o3 framework: a two-agent divide-and-conquer system that decouples visual search from visual reasoning, enabling plug-and-play enhancement of any vReasoner.
Generalized visual search task: extends visual search from specific objects in natural images to relational, fuzzy, or conceptual region descriptions in arbitrary images (maps, infographics, posters).
InSight-o3-vS: an RL-trained vSearcher combining IoU-based out-of-loop training with on-the-fly in-loop training for distribution alignment at inference.
Hybrid sub-agent RL algorithm: novel training regime using both pre-generated tasks (efficiency) and on-the-fly vReasoner-generated tasks (distribution alignment), with global GRPO advantage estimation.

Results¶

O3-Bench: GPT-5-mini improves from 39.0% → 61.5% with InSight-o3-vS as vSearcher.
O3-Bench: OpenAI o3 baseline achieves 40.8% (frontier proprietary, no vSearcher augmentation).
V*-Bench: Gemini-2.5-Flash improves from 80.1% → 87.6% with InSight-o3-vS.
The system "surpasses OpenAI o3 across a variety of benchmarks" per the paper's claim.

Limitations¶

vReasoner is a frozen external model (e.g., GPT-5-mini) at training time; joint training of both agents is left unaddressed due to credit-assignment and non-stationarity challenges.
Benchmark is small (345 QA samples; 204 images), limiting statistical power and domain breadth.
vSearcher is limited to a single image-cropping tool; richer tool sets are noted but not explored.
In-loop RL reward relies on vReasoner ratings (\(s\)) which are self-reported and not always reliable; outcome supervision is used as a corrective but does not eliminate noise.
Training data distribution (synthesized collages + InfographicVQA) differs from O3-Bench evaluation distribution, introducing an unsolved domain gap.

Relevance to Harnesses / Meta-Harnesses¶

InSight-o3 is a concrete multi-agent harness architecture in which a controller agent (vReasoner) dynamically dispatches sub-tasks to a specialized sub-agent (vSearcher) via free-form language interfaces—the defining pattern of a tool-using meta-harness. The hybrid RL training regime, where the sub-agent policy is shaped by both pre-generated offline data and on-the-fly signals from the controller during rollout, directly addresses how to train sub-agents within a harness without joint policy optimization. The plug-and-play design principle—any vReasoner can call any vSearcher without retraining—generalizes the harness composition pattern, and the use of outcome supervision propagated from the orchestrator to the sub-agent is a practical solution to the reward-routing problem central to multi-agent harness training.