GAM-Agent: Game-Theoretic and Uncertainty-Aware Collaboration for Complex Visual Reasoning¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

GAM-Agent wraps arbitrary VLM backbones in a multi-agent orchestration layer that formalizes collaborative reasoning as a non-zero-sum game between specialist base agents and critic agents, mediated by quantified uncertainty. An uncertainty-aware controller dynamically decides when to trigger iterative debate rounds versus accepting the initial integrated response. Evaluated on MMMU, MMBench, MVBench, and V*Bench, it consistently outperforms prior multi-agent debate frameworks across five VLM backbones.

Problem¶

Prior multi-agent VLM systems (MAD, ChatEval, DebUnc, etc.) rely on simple voting or averaging to aggregate agent outputs and lack strategic, evidence-grounded interaction. This makes them brittle on high-complexity visual reasoning tasks (e.g., MMMU) where visual ambiguity and multi-step logical dependencies require targeted dispute resolution. Game-theoretic methods developed for text-based LLM agents cannot transfer directly to the visual domain because they assume reasoning paths composed of textual clues rather than grounded visual evidence.

Method¶

The framework is defined as a six-tuple $S = (E, A, \Phi, M, P, D)$:

Agent roles. $N$ base agents each specialise in one visual subtask (object recognition, scene description, OCR). $N_{crit}$ critical agents (fact checker, completeness checker, logic checker) scrutinise base-agent outputs for factual accuracy, coherence, and completeness.

Uncertainty quantification ($\Phi$). A dual-level estimator is used. When token probabilities are accessible: $$\Phi_i^{gen+}(R_i) = \frac{1}{T_i}\sum_{t=1}^{T_i}\left[\alpha \cdot H(P_{i,t}) + \beta \cdot \max(0, 1 - \Delta_{top}(P_{i,t}))\right]$$ where $H(P_{i,t})$ is token-distribution entropy and $\Delta_{top}$ is the margin between the top-two token probabilities. When probabilities are unavailable, a sigmoid-normalised semantic-marker lexicon estimate $\Phi_i^{sem}(R_i)$ is substituted.

Initial integration and conflict detection. Agents are weighted inversely to uncertainty: $w_i^{(0)} \propto e^{-\beta U_i}$. A debate is triggered iff $U_{sys}^{(0)} > \theta_U$ or $\text{ConflictScore} > \theta_C$ (dual-criterion gate).

Claim Parser (P) and Evidence Mapping (M). Each agent response $R_i$ is parsed into structured tuples $(c_j, \sigma_j, e_j, r_j)$ — claim, confidence, textual evidence, visual-region reference — grounding textual claims to bounding boxes or masks in the image.

Iterative debate. Each round $k$: (1) identify high-conflict/low-confidence claims $C_{debate}^{(k)}$; (2) base agents produce targeted argumentation packages $\text{Arg}_i^{(k)}$; (3) dynamic reweighting by updated $U_i^{(k)}$ or confidence gain $C_i^{(k)}$; (4) IntegrateJudge fuses prior answer, initial responses, structured debate updates, and dispute focus into $R^{(k)}$. Termination when $U_{sys}^{(k)} < \theta_U$, $k \geq K_{max}$, or $\Delta U_{sys}^{(k)} < \epsilon$.

Key Contributions¶

Formalisation of multi-agent VLM collaboration as a non-zero-sum game arbitrated by quantified uncertainty, enabling strategic rather than averaging-based consensus.
Dual-level uncertainty estimator (generation-process entropy/margin + semantic-marker fallback) that works with both open-weight and black-box models.
Claim Parser + Evidence Mapping pipeline that grounds textual agent claims to specific visual regions, enabling fine-grained dispute localisation.
Uncertainty-aware debate controller with dual-criterion triggering and multi-condition termination, avoiding unnecessary debate on unambiguous inputs.

Results¶

Qwen2.5VL-7B (small): +5.11% MMMU, +6.41% MMBench, +6.53% MVBench over base; +1.34% MMBench over best prior (DebUnc).
InternVL3-14B (small): +5.17% MMMU, +6.61% MMBench, +6.64% MVBench — largest absolute gains in the study.
GPT-4o-0513 (large, closed-source): +2.94% MMMU, +3.91% MMBench, +4.04% MVBench over base.
V*Bench (fine-grained visual grounding): Qwen2.5VL-7B +5.25% overall; InternVL3-14B +4.80%; GPT-4o +3.60%; all exceed the reported SEAL baseline (75.39% overall).
GAM-Agent ranks first on every benchmark×backbone combination versus DMAD, DMPL, ChatEval, MAD, and DebUnc.
Larger models (72B/78B) see smaller but consistent gains (+2.7–4.0%), consistent with near-saturation of simpler reasoning subproblems.

Limitations¶

Maximum debate rounds fixed at 3; behaviour under longer chains not reported in main text.
Computational cost scales with $N \times N_{crit} \times K_{max}$; cost-effectiveness analysis deferred to appendix.
Claim Parser and Evidence Mapping depend on the backbone VLM's ability to produce structured outputs; failure modes under low-capability models are not characterised.
Hyperparameters ($\theta_U$, $\theta_C$, $\alpha$, $\beta$, $\gamma$, $\epsilon$) require tuning; sensitivity analysis is in appendix, not main text.
All benchmarks are English-language and academic in nature; generalisation to open-ended tasks or other modalities (audio, video-dense) is untested.

Relevance to Harnesses / Meta-Harnesses¶

GAM-Agent is architecturally a meta-harness: it wraps any VLM backend as a black-box capability and imposes an orchestration layer — uncertainty collection, claim parsing, conflict detection, iterative debate, and termination — without modifying the underlying models. This mirrors the design of harnesses that treat base models as interchangeable components and coordinate them through structured protocols. The dual-criterion debate-trigger (uncertainty threshold $\theta_U$ and conflict score $\theta_C$) is directly analogous to conditional-execution gates in meta-harnesses that decide when to escalate to more expensive sub-pipelines. The modular six-tuple formalism $(E, A, \Phi, M, P, D)$ is a reusable template applicable beyond VLMs — any multi-agent system with heterogeneous specialisation and confidence signals could adopt the same controller structure.