Planning with Quantized Opponent Models¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

QOM (Quantized Opponent Models) compresses the space of opponent policies into a finite discrete catalog via a VQ-VAE, then maintains a Bayesian posterior over those latent types to guide both opponent simulation and agent action selection during Monte-Carlo planning. This unifies type-based modeling, Bayesian inference, and MCTS into a single tractable framework for partially observable multi-agent environments. Experiments across six environments and four baselines show consistent superiority, especially under tight computational budgets.

Problem¶

Planning against unknown opponents in partially observable stochastic games is intractable: type-based methods require hand-crafted behavior classes and fail to scale, while model-free RL is sample-inefficient and cannot principally integrate uncertainty into lookahead search. Particle-filter belief planners (e.g., POTMMCP) degrade under low search budgets because sampling-based inference is expensive relative to the planning horizon.

Method¶

Offline phase: A VQ-VAE encodes opponent trajectories (collected via PSRO policy libraries) into continuous embeddings, then quantizes them to K codebook vectors representing discrete opponent "types." The decoder produces step-wise action likelihoods π̃k(a is estimated by assigning trajectory returns to their type labels.} | h). A payoff matrix R ∈ ℝ^{L×K

Online phase: The agent maintains a categorical belief b_t(k) over K types, updated per observed opponent action via Bayesian rule with temperature β (Eq. 5), plus additive smoothing toward a uniform prior to prevent belief collapse.

Meta-policy: A belief-weighted softmax over per-type soft best-responses (Eq. 2–3) produces π^meta_i, a principled mixture across the agent policy library conditioned on the current posterior.

Planning: At each MCTS simulation step, (i) a type k ~ b_t(k) is sampled to generate opponent actions from π̃_k, (ii) π^meta_i is used as the prior in a PUCT score (Eq. 6), and (iii) the belief is updated at every tree node along the rollout path, not just at the root.

Key Contributions¶

VQ-VAE-based discrete opponent type discovery that replaces manual type definition while remaining interpretable
Online Bayesian belief tracking over latent types with belief smoothing, operating at real-time speeds due to amortized likelihood via the decoder
Belief-weighted meta-policy for rollout action selection, serving as a dynamic PUCT prior
Integration of belief updates within each simulation step (step-level vs. root-level sampling)
Theoretical analysis of posterior concentration under stochastic opponents (referenced in Section A)

Results¶

All comparisons over 300 episodes, varying search time budgets [0.1, 1, 5, 10, 20] units:

Static opponents (6 environments): QOM consistently highest mean return across all search budgets vs. POTMMCP, I-POMCP-PF, and MBOM; advantage sharpest at low search time (0.1–1 units)
Switching opponents (RWS): QOM outperforms all baselines in adapting to piecewise-stationary opponents; particle-filter methods lag behind due to slow posterior updates
Unseen opponents (RWS): QOM generalizes to out-of-distribution policies by assigning mass to nearby latent types; outperforms baselines across all budgets
Scalability (|Π_{-i}| = 30–300): QOM maintains stable performance; particle-based POTMMCP degrades with larger policy libraries due to curse of dimensionality
Ablation – step vs. root sampling: Step-level belief updates (default) consistently outperform root-only sampling
Ablation – meta-policy: Removing meta-policy consistently degrades performance
Ablation – quantization vs. direct: QOM outperforms unquantized exact Bayesian update (50 types) at low budget; performance converges at high budget

Limitations¶

Relies on a fixed, pre-trained policy library; novel opponent behaviors far from the learned codebook reduce performance
Assumes short-term stationarity of opponent policies; abrupt non-stationary switches are handled partially but remain a structural weakness
Requires an environment simulator for offline data generation and online rollouts; applicability to simulator-free settings is unaddressed
Increasing K beyond ~16 causes type redundancy, narrowing posterior informativeness and reducing generalization
Policy libraries built via PSRO require significant offline compute and opponent coverage assumptions

Relevance to Agentic AI / LLM Agents¶

QOM addresses a core challenge for autonomous agents operating in multi-agent settings: how to form and maintain structured beliefs about other agents' hidden intentions without manual specification, and how to exploit those beliefs during lookahead planning. The framework's separation of offline type discovery from online Bayesian inference is directly analogous to capability-learning vs. runtime reasoning patterns emerging in LLM agent architectures. The belief-weighted meta-policy is a compact formalization of "theory of mind" for decision-making agents, a concept increasingly central to multi-agent LLM coordination and adversarial robustness research. This work provides a tractable, principled blueprint for how agents operating under opponent uncertainty can scale belief-aware planning without particle degeneracy.