Skip to content

Wonder Wins Ways: Curiosity-Driven Exploration through Multi-Agent Contextual Calibration

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CERMIC is a plug-and-play MARL exploration module that calibrates each agent's intrinsic curiosity signal using inferred peer intentions, suppressing spurious novelty from stochastic environments while amplifying socially informative signals. It grounds the intrinsic reward in Bayesian Surprise and Information Bottleneck theory, achieving new state-of-the-art on 12/16 tested scenarios across VMAS, MeltingPot, and SMACv2 under sparse rewards.

Problem

Curiosity-driven exploration in MARL suffers from two compounding failures: (1) the "Noisy-TV" problem—agents confuse environmental stochasticity with genuine novelty—and (2) a uniform novelty bias that ignores peer behavioral signals encoding latent task dynamics. Existing mitigations rely on global state or explicit communication, which are unavailable in decentralized, communication-free, partially observable settings.

Method

CERMIC frames exploration as Information Bottleneck optimization over a latent representation \(x_t \sim g_\phi(s_t, a_t)\): maximize \(I(X_t; S_{t+1})\) (novelty-seeking) while minimizing \(I(X_t; [S_t, A_t])\) (compression/noise filtering). The compression term is constrained via a distributionally robust chance constraint conditioned on an inferred multi-agent context feature \(f_n^o\). This feature is produced by a GNN operating on a dynamic intention graph \(G_n\) whose nodes are predicted peer state representations and edges encode spatial relationships; the graph is updated only when peers are detected via a learned MLP, enabling partial-observability efficiency. A task-adaptive calibration factor \(\gamma = I([r_{t-1}^e, f_{n-1}^o]; f_n^o)\)—estimated via InfoNCE—down-weights unreliable peer inferences early in training. The intrinsic reward is a tractable lower-bound Bayesian Surprise: the KL divergence between the conditional latent \(R_\phi(x_t|s_t,a_t)\) and a fixed Gaussian marginal, raised to the \(\frac{1}{2}\) power. In linear MDPs, this reward is provably sandwiched within \([\rho \cdot r_t^{\text{UCB-DB}},\, 2\rho \cdot r_t^{\text{UCB-DB}}]\), guaranteeing decay as uncertainty resolves.

Key Contributions

  • CERMIC framework: socially contextualized curiosity calibration using inferred peer intentions, operable without communication.
  • Distributional robustness formulation: converts the intractable IB compression term into a tractable second-order cone loss via Cantelli's inequality, with both upper and lower bound losses for stability.
  • Task-adaptive calibration factor \(\gamma\): automatically attenuates unreliable intention estimates early in training and misaligned task contexts.
  • Theoretically grounded intrinsic reward: bounded relative to UCB-DB exploration bonus in linear MDPs, ensuring attenuation at convergence.
  • Plug-and-play integration: compatible with MAPPO and QMIX; operates on local observations only.

Results

  • CERMIC achieves SoTA on 12 out of 16 evaluated scenarios across VMAS (9 tasks), MeltingPot (4 tasks), and SMACv2 (2 tasks) under sparse rewards.
  • On MeltingPot (emergent inter-agent reward structure), gains are strongest; e.g., CleanUp: MAPPO-CERMIC 78.3 vs. QPLEX-ICES 76.7 vs. MACE 75.2; ChiGam: 10.03 vs. 9.07 vs. 8.94.
  • Per-agent contribution in Balance* with 8 agents: MAPPO-CERMIC 17.3 vs. MAPPO-DB 12.1—contextual calibration mitigates scaling degradation.
  • Generalization (dense-trained → sparse-tested) performance drop on Balance*: MAPPO-CERMIC −1.7 vs. CPM −2.8 vs. QPLEX-ICES −2.5.
  • Ablation: removing \(L_{\text{explore}}\) degrades Flocki* from 1.06 to 0.82; removing \(L_{\text{exploit}}\) causes variance spike (±0.48 vs. ±0.12).
  • Pretrained intention modules yield faster convergence (exact numbers deferred to appendix).

Limitations

  • Accurate peer intention modeling from local observations alone remains difficult under severe partial observability; early training suffers lower sample efficiency.
  • Dynamic graph construction and one-step latent transition model may not scale to high-dimensional or multi-modal observation spaces without hierarchical extensions.
  • Theoretical guarantees (Theorem 1) are derived only for linear MDPs; nonlinear generalization is empirical.
  • The marginal \(p_{\text{margin}}(x_t|D_m)\) is approximated as a fixed Gaussian \(\mathcal{N}(0,I)\), which may not capture the true data distribution.

Relevance to Agentic AI / LLM Agents

CERMIC addresses a core bottleneck for deploying autonomous agent teams in unstructured environments: how to explore effectively without hand-crafted rewards, explicit communication channels, or centralized state. The insight that agents should calibrate their own curiosity by modeling peer intent—analogous to social learning in developmental psychology—is directly relevant to multi-agent LLM systems where agents must coordinate under partial information. The plug-and-play design and theoretical grounding make CERMIC a candidate primitive for intrinsic motivation in future hybrid neuro-symbolic or LLM-augmented MARL pipelines. The paper also identifies LLMs as a promising bootstrap for the ad-hoc peer intention model, explicitly connecting this MARL work to the LLM agent ecosystem.