Skip to content

GAM-Agent: Game-Theoretic and Uncertainty-Aware Collaboration for Complex Visual Reasoning

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GAM-Agent is a multi-agent framework that models visual reasoning as a non-zero-sum game between specialized base agents (object recognition, scene description, OCR) and critical agents (fact-checker, logic-checker, completeness-checker), arbitrated by quantified uncertainty. An uncertainty-aware controller triggers iterative debates only when agent disagreement or ambiguity exceeds thresholds, yielding adaptive and interpretable consensus. It consistently outperforms prior multi-agent debate baselines across four VLM benchmarks and five backbone models.

Problem

Existing multi-agent VLM debate frameworks rely on naive aggregation (voting/averaging) without strategic agent interactions, making them brittle in high-complexity visual reasoning scenarios (e.g., MMMU). Pure LLM game-theoretic methods depend heavily on text-based reasoning paths and cannot be directly applied to vision-language settings where uncertainty arises from visual ambiguity (occlusions, spatial grounding failures).

Method

GAM-Agent is formalized as a six-tuple S = (E, A, Φ, M, P, D). Two agent cohorts operate in parallel: - Base Agents produce initial visual interpretations from distinct perspectives (object recognition, scene description, OCR). - Critical Agents evaluate factual accuracy, logical coherence, and completeness of base agent outputs.

Agent outputs are parsed into structured claim tuples (claim, confidence, textual evidence, visual region reference) via a Claim Parser (P), then grounded to image regions via Evidence Mapping (M).

Uncertainty is quantified by a dual-level function Φ: (1) generation-process uncertainty using token entropy and top-2 probability gap when logits are accessible, or (2) semantic-marker-based uncertainty via a weighted lexicon otherwise.

A Debate Controller computes weighted-average system uncertainty U_sys and a conflict score. If either exceeds thresholds θ_U or θ_C, it triggers iterative debate rounds in which agents refine targeted arguments around identified dispute focal points. Agent weights are dynamically adjusted per-round by inverse-exponential uncertainty. Debate terminates when U_sys < θ_U, max rounds (K_max) is reached, or uncertainty change Δ_sys < ε.

Key Contributions

  • Game-theoretic formulation of multi-agent VLM reasoning as a non-zero-sum game with structured claim-evidence interactions
  • Dual-level uncertainty quantification (generation-process and semantic-marker-based) applicable when model logits are or are not accessible
  • Uncertainty-triggered adaptive debate: debates are skipped for easy cases, invoked iteratively for hard ones, avoiding unnecessary computation
  • Evidence Mapping module that grounds textual claims to specific visual regions, enabling precise conflict localization during debate
  • Modular, backbone-agnostic framework demonstrated on 5 VLMs (Qwen2.5VL-7B/72B, InternVL3-14B/78B, GPT-4o-0513)

Results

  • Qwen2.5VL-7B on MMBench_V11_Test: +6.41% (82.61→89.02%) vs. best baseline DebUnc +5.04%
  • InternVL3-14B on MMBench_V11_Test: +6.61% (83.54→90.15%) vs. DebUnc +5.32%
  • InternVL3-14B on MVBench_Test: +6.64% (76.59→83.23%)
  • GPT-4o-0513 on MMBench_V11_Test: +3.91% (83.13→87.04%)
  • GPT-4o-0513 on MMMU: +2.94% (68.97→71.91%)
  • V*Bench (Qwen2.5VL-7B): +5.25% overall (61.26→66.51%); GAM-Agent with GPT-4o-0513 reaches 78.32% vs. SEAL's 75.39%
  • Ablation: removing iterative debate costs −4.28% accuracy; removing claim/evidence parsing costs −4.72% and causes 80% debate trigger rate; full system achieves 88.80% at 1.76 avg debate rounds (lowest cost)
  • Optimal performance saturates at 3–4 max debate rounds; actual rounds average 1.76 due to early termination

Limitations

  • Inference cost scales with number of agents and debate rounds; production deployment requires careful cost-accuracy trade-offs (studied in appendix but not detailed in main text)
  • Game-theoretic formulation is non-zero-sum cooperative but lacks formal equilibrium guarantees; convergence relies on empirical thresholds θ_U, θ_C, ε that require tuning per task/model
  • Evaluation confined to multiple-choice VQA-style benchmarks; open-ended generation tasks not assessed
  • Evidence Mapping relies on the base VLMs' own grounding capability; no external visual grounding model is introduced, so it inherits backbone grounding limitations
  • All experiments use N=3 base and N=3 critic agents; scalability with more agents is not systematically studied

Relevance to Agentic AI / LLM Agents

GAM-Agent advances a principled alternative to naive multi-agent voting/debate by introducing game-theoretic interaction and uncertainty quantification as first-class coordination primitives—directly applicable to any agentic system requiring multi-agent consensus under uncertainty. The uncertainty-triggered debate controller is a concrete mechanism for adaptive resource allocation in agent pipelines, invoking expensive collaboration only when needed—a pattern transferable to LLM-only agent systems. The claim-evidence-region traceability pipeline offers a blueprint for interpretable multi-agent reasoning chains, addressing the common opacity problem in agent collaboration. This work sits at the intersection of multi-agent debate (Du et al., 2023), uncertainty-aware reasoning, and VLM agent systems, extending the frontier toward reliable multimodal agentic reasoning.