Be My Eyes: Extending Large Language Models to New Modalities Through Multi-Agent Collaboration

🕒 Published (v1): 2025-11-24 18:55 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

BeMyEyes is a multi-agent framework that extends frozen, text-only LLMs to multimodal reasoning by pairing them with small, fine-tuned VLMs acting as perceiver agents. The perceiver extracts and communicates visual information through multi-turn conversation; the reasoner (LLM) does all complex inference. With a Qwen2.5-VL-7B perceiver, DeepSeek-R1 outperforms GPT-4o on three of four benchmarks without any modification to the LLM.

Problem

Training or fine-tuning large-scale multimodal models (VLMs) to add visual perception to capable LLMs is computationally expensive, modality-specific, and requires architectural changes. Smaller VLMs are efficient but lack the knowledge and reasoning depth of frontier LLMs. No existing framework enables modular, low-cost modality extension that preserves LLM generalization while allowing independent swapping of perception and reasoning components.

Method

BeMyEyes instantiates two agents: a perceiver (small VLM, e.g., Qwen2.5-VL-7B or InternVL3-8B) and a reasoner (frozen LLM, e.g., DeepSeek-R1). Only the perceiver receives visual input; the reasoner operates text-only. An orchestration layer initializes both with role-aware system prompts and manages up to 5 alternating turns: the reasoner requests visual details; the perceiver responds with grounded descriptions; the perceiver produces the final answer after deliberation.

To adapt the perceiver for this collaborative role, a data synthesis pipeline generates 12,145 training instances from CoSyn-400K images: (1) GPT-4o generates reasoning-intensive questions; (2) GPT-4o role-plays both agents to simulate BeMyEyes conversations, sampling up to 8 per question and retaining those that yield the correct answer; (3) instances answerable without vision or with no successful conversation are filtered. The perceiver is then fine-tuned via supervised learning on its own turns within these conversations:

\[\mathcal{L}_{\text{SFT}} = -\sum_{t=1}^{T} \log P_\theta(y_t \mid y_{<t}, x)\]

The reasoner requires no fine-tuning or architectural modification.

Key Contributions

• Multi-agent orchestration framework decoupling perception (small VLM) from reasoning (frozen LLM) via structured multi-turn dialogue.
• Data synthesis pipeline that generates perceiver training data by having a teacher model (GPT-4o) simulate the full harness interaction protocol.
• Supervised fine-tuning strategy that improves perceiver collaboration behavior without meaningfully improving its standalone VLM accuracy.
• Demonstrated modular extensibility to specialized domains (medical imaging via Lingshu-7B) without domain-specific training data.

Results

• DeepSeek-R1 + BeMyEyes (Qwen2.5-VL-7B) vs. GPT-4o: +0.5% MMMU Pro (57.2 vs. 56.7), +7.1% MathVista (72.7 vs. 65.6), +12.1% MathVision (48.5 vs. 36.4); MMMU slightly below (67.4 vs. 68.3).
• DeepSeek-R1 + BeMyEyes (InternVL3-8B): outperforms GPT-4o on all four benchmarks (MMMU 69.7, MMMU Pro 58.5, MathVista 73.1, MathVision 50.6 vs. 68.3/56.7/65.6/36.4).
• Medical domain (Lingshu-7B perceiver): DeepSeek-R1 reaches 76.7/58.7 on MMMU Med / MMMU Pro Med vs. GPT-4o's 78.0/59.1, without medical-specific training data.
• Ablation: removing SFT drops MMMU Pro from 49.9 to 49.1; single-turn drops it to 48.3; standalone SFT on the perceiver yields minimal gains (54.0→54.6 MMMU, 39.8→40.6 MMMU Pro).

Limitations

• Data synthesis depends on GPT-4o (closed, expensive); quality degrades with weaker teacher models.
• Sampling budget of 8 conversations per question; questions requiring highly abstract visual communication are discarded, creating selection bias in what the framework can handle.
• Text-based visual communication is inherently lossy; some visual information (complex diagrams, fine-grained spatial detail) cannot be conveyed reliably through natural language descriptions.
• Multi-agent dynamics occasionally cause the reasoner to be misled by perceiver errors, creating failure modes absent in single-model settings.
• Reasoning traces capped at 4096 tokens for thinking models, potentially truncating complex chains.

Relevance to Harnesses / Meta-Harnesses

BeMyEyes is a textbook agent harness: it defines a fixed interaction protocol (role-typed system prompts, turn limits, answer extraction) that coordinates two heterogeneous agents without modifying either's weights at inference time. More distinctively, the data synthesis pipeline constitutes a meta-harness pattern — GPT-4o simulates the harness's own interaction protocol to generate training signal for a component of that harness, effectively using the harness structure as its own data generator. The ablation result that SFT improves collaboration rather than standalone capability is direct empirical evidence that harness-aware training is qualitatively distinct from standard fine-tuning. This work is directly relevant to harness designers asking how to train sub-agents to behave correctly within a fixed orchestration contract.