Skip to content

GAM-Agent: Game-Theoretic and Uncertainty-Aware Collaboration for Complex Visual Reasoning

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GAM-Agent frames multi-VLM collaboration as a non-zero-sum game between specialised Base Agents (object recognition, scene description, OCR) and Critical Agents (fact-checking, logic-checking, completeness), mediated by a quantified uncertainty signal that triggers iterative debate only when needed. The framework is plug-and-play over any VLM backbone and consistently outperforms prior multi-agent debate methods across four benchmarks.

Problem

Existing multi-agent debate frameworks for VLMs use naive aggregation (majority voting, simple averaging) without strategic interaction or uncertainty awareness, causing them to underperform on high-complexity visual reasoning tasks (e.g., MMMU). Game-theoretic collaboration methods from the LLM literature rely on text-only reasoning chains and cannot be directly applied to vision-language inputs.

Method

GAM-Agent is defined as a six-tuple S = (E, A, Φ, M, P, D). Two agent tiers operate in parallel: Base Agents (object recognition, scene description, OCR) generate initial analyses, while Critical Agents (fact checker, completeness checker, logic checker) scrutinise those outputs. A Claim Parser (P) decomposes unstructured responses into structured tuples (claim, confidence, textual evidence, visual region reference), and an Evidence Mapping module (M) grounds each claim to a bounding-box or described image region.

Uncertainty is quantified via a dual-level function Φ: (1) generation-process uncertainty Φ^gen combines token-distribution entropy H(P_{i,t}) and the top-2 probability gap Δtop when logits are accessible; (2) semantic-marker uncertainty Φ^sem applies a weighted lexicon of hedging markers when logits are unavailable, normalised via sigmoid. Agent initial weights are set as w_i^(0) ∝ exp(−βU_i).

A Debate Controller computes system uncertainty U_sys^(0) and an inter-expert ConflictScore; debate is triggered when either exceeds a threshold. Each debate round: (a) identifies disputed claims with low confidence or high inter-agent conflict, (b) each Base Agent generates targeted argumentation, (c) Critical Agents evaluate, (d) weights are updated by exp(−βU_i^(k)) or exp(γC_i^(k)), and (e) IntegrateJudge fuses everything into R^(k). Termination occurs when U_sys^(k) < θ_U, Δ U_sys^(k) < ε, or k ≥ K_max (empirically 3 rounds suffice).

Key Contributions

  • Game-theoretic formulation of multi-VLM collaboration as a non-zero-sum debate between Base and Critical agent cohorts.
  • Dual-level uncertainty quantification (token-distribution entropy + semantic marker fallback) usable with both open-weight and API-only VLMs.
  • Claim Parser + Evidence Mapping pipeline that grounds textual claims to specific visual regions, enabling fine-grained conflict localisation.
  • Uncertainty-aware debate controller that skips debate on easy examples, reducing average inference cost.
  • Modular, backbone-agnostic architecture validated across five VLMs (7B–78B open-source, GPT-4o).

Results

  • Qwen2.5-VL-7B on MMMU: +5.11% (53.82→58.93%); MMBench: +6.41% (82.61→89.02%); MVBench: +6.53%.
  • InternVL3-14B on MMBench: +6.61% (83.54→90.15%); MVBench: +6.64%.
  • GPT-4o-0513 on MMBench: +3.91% (83.13→87.04%); MMMU: +2.94%.
  • V*Bench (fine-grained visual grounding): Qwen2.5-VL-7B +5.25% overall; GPT-4o-0513 78.32% vs. SEAL reference 75.39%.
  • GAM-Agent consistently leads all five baselines (DMAD, DMPL, ChatEval, MAD, DebUnc) on every model-benchmark combination reported.
  • Ablation: removing iterative debate costs −4.28% accuracy; removing P&M costs −4.72% and raises debate trigger rate to 80%; removing uncertainty quantification increases debate length and cost.
  • Max debate rounds study: accuracy plateaus after 3 rounds (86.53%); average actual rounds stay below the maximum due to early termination.

Limitations

  • Inference cost scales with number of agents (3 base + 3 critical) and debate rounds; not profiled for latency-sensitive deployments.
  • Hyper-parameters (β, γ, θ_U, θ_C, ε) require tuning and are not self-calibrated.
  • Evidence Mapping relies on the base VLM's ability to identify bounding boxes/regions; weak grounding models may produce unreliable visual evidence links.
  • All experiments capped at max 3 debate rounds and N=3 agents; scaling behaviour is not fully characterised.
  • Gains on MMMU are smaller (~1–3%) than on MMBench/MVBench, suggesting the framework is less effective on densely knowledge-dependent tasks.

Relevance to Vision-Language Models

GAM-Agent directly addresses a known weakness of VLMs—single-pass visual reasoning under ambiguity—by introducing structured inter-agent debate grounded in quantified uncertainty, which is more principled than the majority-vote or simple-averaging schemes common in prior VLM ensembles. The plug-and-play design means any backbone (open or closed, small or large) can benefit without retraining, making it immediately applicable alongside current SOTA VLMs like Qwen2.5-VL and InternVL3. The evidence-grounding pipeline (Claim Parser + Evidence Mapping) adds an interpretability layer that is largely absent from existing VLM multi-agent work. For researchers tracking VLMs, this paper demonstrates that inference-time collaboration with uncertainty-aware control is a practical alternative to scaling model size for improving complex visual reasoning.