DART: Leveraging Multi-Agent Disagreement for Tool Recruitment in Multimodal Reasoning¶

🕒 Published (v1): 2025-12-08 03:33 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

DART is a multi-agent VQA framework that uses inter-agent disagreement as a signal to trigger targeted recruitment of specialized vision tools (grounding, OCR, spatial reasoning, etc.), then uses tool-aligned agreement scores to weight agent contributions during discussion and aggregation. By combining VLM debate diversity with expert-tool precision, DART consistently outperforms both pure multi-agent debate and single-agent tool-calling baselines across four benchmarks.

Problem¶

Multi-agent debate among VLMs stalls because agents share the same perceptual blind spots (e.g., weak OCR, grounding, spatial reasoning) and lack new information to break deadlocks. Conversely, single-LLM tool-calling systems plan tool usage top-down from the question alone, ignoring visual content and offering no diverse reasoning paths. Neither paradigm alone resolves perceptual disagreements productively.

Method¶

DART operates in five sequential steps over a set of VLM answering agents and a pool of vision expert tools:

Initial Answer Generation — each VLM agent produces an answer, CoT reasoning chain, and self-reported confidence; outputs are grouped by unique answer with vote counts.
Tool-Based Disagreement Resolution — a dedicated recruitment agent (Qwen2.5-7B) reads the grouped answers, identifies specific perceptual conflicts (e.g., color discrepancy), and selects which tools to invoke with what query parameters. Tools include GroundingDINO, YOLOv11, SpaceLLaVA, OCR-Qwen, and InternVL-2.5 (captioning/attribute/reasoning).
Tool-Based Agreement Scoring — for each (agent output, tool output) pair, a scorer LLM produces a binary agreement score \(s_{ij} \in \{0,1\}\); the per-agent score \(S_i\) is the mean across all invoked tools, serving as an expert-calibrated confidence proxy.
Discussion — agents receive the grouped initial answers, tool outputs, and agreement scores; they are instructed to prioritize expert tool outputs over peer claims and refine their answers accordingly.
Aggregation — an aggregator VLM (Ovis2) selects the final answer via CoT using post-discussion grouped answers, tool outputs, and recalculated agreement scores.

Tool diversity is enforced by using a separate VLM (InternVL-2.5) for general-purpose tool tasks to avoid answer-agent overlap; specialized fine-tuned models handle grounding, detection, OCR, and spatial tasks.

Key Contributions¶

Disagreement-triggered tool recruitment: tools are called only on the specific perceptual conflicts detected in the debate, not planned top-down from the question.
Tool-aligned agreement scoring as a calibrated confidence signal, replacing or supplementing self-reported agent confidence.
Empirical demonstration that multi-model agent diversity (vs. multi-instance same-model) is essential—heterogeneous agents produce 2.7× more disagreements, enabling more tool invocations.
Plug-and-play domain adaptation: adding a single domain-specific tool (MedGemma 4B) to the pool suffices to extend DART to medical VQA without retraining.
Quantitative evidence that DART discussions are substantially more novel than standard debate (ROUGE-L drops 0.173, Jaccard drops 0.289 vs. multi-agent debate).

Results¶

A-OKVQA: DART (multi-model) 68.9% vs. best multi-agent baseline (Debate with Judge, Q+M+O) 65.5% (+3.4%); vs. ViperGPT 49.9% (+19.0%); vs. Chameleon 47.5% (+21.4%).
MMMU: DART 65.5% vs. Debate with Judge (Q+M+O) 63.1% (+2.4%); vs. best single-agent CoT (QwenVL) 58.8%.
NaturalBench: DART 82.5% vs. Debate with Judge (Q+M+O) 81.3% (+1.2%).
M3D medical VQA: DART 57.8% avg. vs. QwenVL (best single-agent) 56.4% (+1.4%); vs. Debate with Judge 53.4% (+4.4%); naive multi-agent debate hurts by −2.7% vs. single-agent, while DART recovers and exceeds.
Ablations (A-OKVQA): removing diverse answerers −3.6%; removing expert tool outputs −3.2%; removing agreement scores −1.1%.
Tool mix: replacing all tools with a single generalist VLM costs −2.9%; removing only specialized model tools costs −3.3%; removing only VLM-based tools costs −2.2%.

Limitations¶

Latency/cost scales with number of agents, tools invoked, and discussion rounds; no efficiency analysis provided in the main paper.
Tool pool must be manually curated per domain; DART does not learn which tools to add or synthesize new ones.
Recruitment agent depends on textual descriptions of tool capabilities; tools with poorly described interfaces may be under-recruited or misused.
Evaluated only on VQA tasks; generalization to open-ended generation or sequential decision tasks is untested.
Uses fixed agent counts (three) and a single debate round in main experiments; scalability to more rounds or agents is deferred to appendices.
All VLM agents share the same architectural family of prompt-following models; adversarial or safety-constrained domains may behave differently.

Relevance to Harnesses / Meta-Harnesses¶

DART is a concrete instance of a meta-harness: a coordinator layer that dynamically dispatches to specialized sub-agents (tools) based on runtime signals (disagreement) rather than static planning. The five-step pipeline—generate, detect conflict, recruit tools, score agreement, aggregate—is a structured orchestration harness with explicit inter-step dependencies, directly analogous to meta-harness patterns that route tasks to specialist modules based on uncertainty or failure signals. The disagreement-triggered recruitment mechanism is particularly relevant as a principled trigger design for harnesses: instead of always calling all tools or relying on a planner's prior, the harness invokes capabilities on-demand where they are most likely to add value. The plug-and-play domain extension result (one new tool → new domain) demonstrates modular composability, a key property for maintainable meta-harness architectures.